hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2168) Pipes with a C++ record reader does not update progress in the map until it is 100%
Date Wed, 13 Aug 2008 09:14:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622142#action_12622142

Arun C Murthy commented on HADOOP-2168:

This is turning out to be harder than I expected - the problem being that only MapTask.TrackerRecordReader
updates progress and in this case the Pipes application isn't looping through the Java RecordReader...

Possible solutions:
1. Add a 'setProgress' api to org.apache.hadoop.mapred.Reporter - this is quite dangerous
since applications can potentially use this to throw the Scheduler off.
2. Use a dummy InputFormat/RecordReader along with a specialized Reporter which can communicate
the progress to the RecordReader and hence the rest of the framework.

Given the dangers of the first approach, Owen/me agree we should go with the second for now.

> Pipes with a C++ record reader does not update progress in the map until it is 100%
> -----------------------------------------------------------------------------------
>                 Key: HADOOP-2168
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2168
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: pipes
>            Reporter: Owen O'Malley
>            Assignee: Arun C Murthy
> Currently C++ maps with C++ record readers do not update the progress, even if the application
has updated it.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message