hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3748) Flag to make tasks to send counter information only at the end of the task
Date Fri, 29 Aug 2008 11:31:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626938#action_12626938

Alejandro Abdelnur commented on HADOOP-3748:

Our test was focused on seeing how jobs with a large number of counters can affect the cluster.

Our observations indicate that number of network pkg_in traffic increases significantly and
faster than linear (we are now testing on a 400 nodes to confirm this).

The test MR job itself was not doing any thing useful and we can claim performance improvement
on job execution as our test map is coded to run for 30 mins (write counters, sleep, ...).

The reasoning is that by lowering (significantly) the network traffic to the JT we are making
things easier for it, besides less network overhead.

BTW, the path would be very simple, ~15 lines of code that just add an {{IF}} condition in
a couple of places.

> Flag to make tasks to send counter information only at the end of the task
> --------------------------------------------------------------------------
>                 Key: HADOOP-3748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3748
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
> Currently counters are streaming from the task to the jobtracker as the task progresses.
If the number of counters is large this has a significant impact on the network traffic as
well as in the JobTracker load.
> The should be a flag, for example by counter-group, that indicates that the counters
are to be reported at the end of the task. By default this flag should be set to false for
all counter-groups maintaining the current behavior.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message