hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lohit (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1210) Log counters in job history
Date Thu, 23 Aug 2007 17:52:30 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

lohit updated HADOOP-1210:
--------------------------

    Attachment: h-1210-jobstatus.patch

This looks like useful feature to have. As to get it started, I am uploading an initial patch
which will dump the counters onto standard out via hadoop job -status command. The output
would be similar to the one shown below. Suggestions/Inputs as to what could be changes? The
getCounters() API works great for global counters. Is there a way to fetch counters from Individual
map/reducers?

Job: job_200708231653_0002
file: /hadoop/mapred/system/job_200708231653_0002/job.xml
tracking URL: http://myhost:50030/jobdetails.jsp?jobid=job_200708231653_0002
map() completion: 1.0
reduce() completion: 1.0
Job Counters:
  Counter Group : org.apache.hadoop.mapred.JobInProgress$Counter
        TOTAL_LAUNCHED_MAPS = 2
        TOTAL_LAUNCHED_REDUCES = 1
        DATA_LOCAL_MAPS = 2
  Counter Group : org.apache.hadoop.mapred.Task$Counter
        MAP_INPUT_RECORDS = 1967
        MAP_OUTPUT_RECORDS = 5288
        MAP_INPUT_BYTES = 74377
        MAP_OUTPUT_BYTES = 84327
        COMBINE_INPUT_RECORDS = 5288
        COMBINE_OUTPUT_RECORDS = 2293
        REDUCE_INPUT_GROUPS = 2253
        REDUCE_INPUT_RECORDS = 2293
        REDUCE_OUTPUT_RECORDS = 2253



> Log counters in job history
> ---------------------------
>
>                 Key: HADOOP-1210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1210
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Albert Chern
>            Priority: Minor
>         Attachments: h-1210-jobstatus.patch
>
>
> It would be useful if the value of the global counters were logged to the job history,
perhaps even individually for each task after completion.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message