hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2284) BasicTypeSorterBase.compare calls progress on each compare
Date Thu, 27 Dec 2007 02:35:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554508

Arun C Murthy commented on HADOOP-2284:

bq. Currently time-based seems like a better technique. Comments?

I agree, however it should be a fraction of {{mapred.task.timeout}} (say 10%), else we run
the risk of the tasks being timed-out by the {{TaskTracker}}.

> BasicTypeSorterBase.compare calls progress on each compare
> ----------------------------------------------------------
>                 Key: HADOOP-2284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2284
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
> The inner loop of the sort is calling progress on each compare. I think it would make
more sense to call progress in the sort rather than the compare or at most every 10000 compares.
In the performance numbers, the call to progress as part of the sort are consuming 12% of
the total cpu time when running word count under the local runner.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message