hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4714) map tasks timing out during merge phase
Date Mon, 24 Nov 2008 07:29:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650126#action_12650126
] 

Jothi Padmanabhan commented on HADOOP-4714:
-------------------------------------------

It could also be possible that we have lesser records in each spill file because each record
is of a bigger size. Since the limits for spill files are by size (io.sort.mb) and not record
count, we probably are not touching the 10,000 mark at all?

Christian, could you tell us what would be the approximate/average record size of the intermediate
map output and the io.sort.mb in this case?


> map tasks timing out during merge phase
> ---------------------------------------
>
>                 Key: HADOOP-4714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4714
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.1
>            Reporter: Christian Kunz
>
> With compression of transient data turned on some parts of the merge phase seem to not
report progress enough.
> We see a lot of task failures during the merge phase, most of them timing out (even with
a 20 min timeout)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message