hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5876) Shuffling information logged to userlogs/attempt_####_###_r_###_#/syslogs
Date Wed, 20 May 2009 22:00:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711384#action_12711384

Eric Yang commented on HADOOP-5876:

Although shuffling information is available in hadoop metrics form.  It doesn't contain the
full information.  For example:

Timestamp : 1242815580000
[jobId] :job_200905200055_0930
[jobName] :Chukwa-Demux_20090520_10_32
[recordName] :shuffleInput
[sessionId] :
[shuffle_failed_fetches] :0
[shuffle_fetchers_busy_percent] :0
[shuffle_input_bytes] :10
[shuffle_success_fetches] :5
[taskId] :attempt_200905200055_0930_r_000006_0

The task attempt id doesn't indicate the source of the shuffle.  It is difficult to match
the corresponding shuffle input and output.  In addition, the start time and end time of the
suffle would also be useful.  What is the easiest way to get this information into Chukwa?

> Shuffling information logged to userlogs/attempt_####_###_r_###_#/syslogs
> -------------------------------------------------------------------------
>                 Key: HADOOP-5876
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5876
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Priority: Critical
> Shuffling information is currently logged to userlogs, it would be ideal to have this
information consolidated in task tracker log or job history log file for down stream log collection
and analysis program (Chukwa) to pickup.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message