hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1042) Improve the handling of failed map output fetches
Date Tue, 27 Feb 2007 07:27:05 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Devaraj Das updated HADOOP-1042:

    Attachment: 1042.patch

This patch does the following (everything in the file ReduceTaskRunner.java):
1) Changes the datastructure of knownOutputs from List to Map. This eases replacing MapOutputLocation
objects for the failed fetches (if the JobTracker later on gives us new locations for those
2) Changes ListIterator to Iterator (since it is not straightforward to get a ListIterator
out of a Map and we anyway don't use the features of a ListIterator)
3) Changes the order in which entries (mapId/MapOutputLocation objects) are added in the knownOutputs
Map - first entries corresponding to failed fetches are added and then the new entries (got
from JobTracker) are added. This will ensure that the new entries overwrite the old (failed)
entries (for the same mapId hashkeys).
4) Removes the call to Collections.shuffle( ) and the associated Random object. Since the
randomness for fetching map outputs is not there anymore, we don't need this.
5) queryJobTracker now returns a List <MapOutputLocation> instead of an array of MapOutputLocation.

> Improve the handling of failed map output fetches
> -------------------------------------------------
>                 Key: HADOOP-1042
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1042
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.11.2
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 1042.patch
> Currently, whenever fetch of a map output fails the corresponding MapOutputLocation is
added to a List datastructure for later retrial. But, if the failure was due to a lost task,
the entry that was added is not deleted. For such cases, unnecessary retrials will happen.
This situation should be prevented.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message