hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-318) Progress in writing a DFS file does not count towards Job progress and can make the task timeout
Date Fri, 23 Jun 2006 18:18:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-318?page=comments#action_12417539 ] 

Milind Bhandarkar commented on HADOOP-318:
------------------------------------------

I don't see how this can be done without breaking backward-compatibility. Therefore I have
made changes so that with minimum porting any other output formats could be incorporated.
An additional getRecordWriter method needs to be implemented that takes an additional parameter.
This parameter can be passed to fs.create (or even ignored as in the case of local filesystem.)


> Progress in writing a DFS file does not count towards Job progress and can make the task
timeout
> ------------------------------------------------------------------------------------------------
>
>          Key: HADOOP-318
>          URL: http://issues.apache.org/jira/browse/HADOOP-318
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.3.2
>  Environment: all, but especially on big busy clusters
>     Reporter: Milind Bhandarkar
>     Assignee: Milind Bhandarkar
>      Fix For: 0.4.0
>  Attachments: hadoop-latency-new.patch, hadoop-latency.patch
>
> When a task writes to DFS file, depending on how busy the cluster is, it can timeout
after 10 minutes by default, because the progress towards writing a DFS file does not count
as progress of the task. The solution (patch is forthcoming) is to provide a way to callback
reporter to report task progress from DFSOutputStream.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message