hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Bowen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1485) Metrics should be there for reporting shuffle failures/successes
Date Fri, 22 Jun 2007 19:46:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507489
] 

David Bowen commented on HADOOP-1485:
-------------------------------------


Reviewing 1485.1.patch.

Two classes with the name ShuffleMetrics is confusing.  Please rename at least one of them,
and add some per-class comments explaining their purposes.

TaskTracker.ShuffleMetrics:

   * shuffle_handler_busy_percent seems to be an absolute value, i.e. it should be using setMetric
rather than incrMetric.  Also, shuffle_failed_outputs and shuffle_success_outputs seem to
be relative values, and so should be using incrMetric rather than setMetric.
   * It may be an unnecessary optimization, but it couldn't hurt to move the shuffleMetricsRecord.update
call out of the synchronized block.  update() has to do a little bit of work, and there's
no need to be holding the lock.
   * MapOutputServlet is missing indentation under the first "try {".
   * The final finally may need to call shuffleMetrics.update.

ReduceTask.ReduceCopier.ShuffleMetrics:

   * I think incrMetric should be being used for shuffle_failed_fetches and shuffle_success_fetches.
   * Same comment about moving the shuffleMetrics.update() call out of the synchronized block.




> Metrics should be there for reporting shuffle failures/successes
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1485
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1485
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.14.0
>
>         Attachments: 1485.1.patch, shuffle-metrics.patch
>
>
> It would be nice to have metrics for the shuffle phase which reports the failures/successes
for the fetches. This would aid in performance tests and in debugging (shuffle).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message