Configure hadoop job failure percent

 

For some applications, it is undesirable to abort the job when few tasks fail, as it may be possible to use the results of the job despite of some failures.

In this case the maximum percentage of tasks that are allowed to fail without triggering job failure can be set for the job.

Map tasks are controlled by using mapred.max.map.failures.percent property. If we set this value as 50, map tasks will get finished even though 50% of tasks are killed without failing the job.

Reduce tasks are controlled by using property mapred.max.reduce.failures.percent property. If we set this value as 30, reduce tasks will get finished even though 30% of tasks are killed without failing the job.

-Sany

Advertisements
By Sandeep Posted in Hadoop

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s