hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3748) Flag to make tasks to send counter information only at the end of the task
Date Tue, 02 Sep 2008 16:35:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-3748:
----------------------------------

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

I think this is a really bad plan. The application can already do this by keeping the counters
local until the map or reduce close method is called. We already have broken code in the task
that tries to only send counters occasionally that should be removed.

> Flag to make tasks to send counter information only at the end of the task
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-3748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3748
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
>         Attachments: [HADOOP-3748].patch
>
>
> Currently counters are streaming from the task to the jobtracker as the task progresses.
If the number of counters is large this has a significant impact on the network traffic as
well as in the JobTracker load.
> The should be a flag, for example by counter-group, that indicates that the counters
are to be reported at the end of the task. By default this flag should be set to false for
all counter-groups maintaining the current behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message