hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3748) Flag to make tasks to send counter information only at the end of the task
Date Tue, 02 Sep 2008 11:31:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627633#action_12627633
] 

Alejandro Abdelnur commented on HADOOP-3748:
--------------------------------------------

We've run a test in a 400 nodes cluster and the network pkg_in/min increase in the JT is 66%.

Consolidating numbers:

|| Cluster size || pkg_in Idle || pkg_in wo/counters || pkg_in w/counters || pkg_in % increase
|| pkg_in % increase excluding idle load ||
| 100 | 0654 | 1080 | 1380 | 28% | 70% |
| 200 | 0397 | 0591 | 0945 | 60% | 182% |
| 400 | 1660 | 1840 | 3070 | 67% | 783% |

* pkg_in are per minute.
* network load in the 100 node cluster was consistently higher even in idle mode.

As I've mentioned before our tests were focusing on finding the overhead of using 200 counters
not on overall performance impact. The test jobs where not creating any input/output data
network traffic.


> Flag to make tasks to send counter information only at the end of the task
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-3748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3748
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
>         Attachments: [HADOOP-3748].patch
>
>
> Currently counters are streaming from the task to the jobtracker as the task progresses.
If the number of counters is large this has a significant impact on the network traffic as
well as in the JobTracker load.
> The should be a flag, for example by counter-group, that indicates that the counters
are to be reported at the end of the task. By default this flag should be set to false for
all counter-groups maintaining the current behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message