hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4714) map tasks timing out during merge phase
Date Sun, 30 Nov 2008 02:24:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Douglas updated HADOOP-4714:
----------------------------------

    Status: Open  (was: Patch Available)

bq. Attaching a patch that uses a single recordCounter per task. This counter is used while
merging and is not reset per reducer 
Passing yet another counter to the merge seems unnecessary; sharing an AtomicLong between
counters to maintain the arbitrary, 10k threshold is pressing a heuristic into service as
an API.

Why not simply report progress at least once for each partition during the merge?

> map tasks timing out during merge phase
> ---------------------------------------
>
>                 Key: HADOOP-4714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4714
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.1
>            Reporter: Christian Kunz
>            Assignee: Jothi Padmanabhan
>         Attachments: hadoop-4714.patch
>
>
> With compression of transient data turned on some parts of the merge phase seem to not
report progress enough.
> We see a lot of task failures during the merge phase, most of them timing out (even with
a 20 min timeout)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message