hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4693) Historyserver should provide counters for failed tasks
Date Wed, 20 Feb 2013 23:45:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582675#comment-13582675
] 

Xuan Gong commented on MAPREDUCE-4693:
--------------------------------------

bq:TaskAttempt20LineEventEmitter, Task20LineHistoryEventEmitter don't need to be changed -
unless this change is being made in branch-1 as well.

I did some changes on TaskFailedEvent and TaskAttemptUnsuccessfulCompletionEvent, add new
constructer without counter parameter. Otherwise, there will be error on these two TaskAttempt20LineEventEmitter,
Task20LineHistoryEventEmitter since we have added counter as new parameter at the old patch.

bq:JobBuilder should be able to handle null counters.

At the new patch, handle the null counters, use EMPTY_COUNTERS when the counters is null.

bq:TaskFailedEvent and TaskAttemptUnsuccessfulCompletionEvent should store counters as 
org.apache.hadoop.mapreduce.Counters, and convert to jobhistory.JhCounters only while serializing.
(See MapAttemptFinishedEvent). That's to lower the AM memory overhead in case the history
events processor falls behind.

I think this has already been handled. Already made changes on Event.avro, and the TaskFailed
and TaskAttemptUnsuccessfulCompletion will be automatically generated by avro, and the counters
is converted to jobhistory.JhCounters while serializing.

bq:Needs a unit test.

We already have a testcase to test it, I made a simple change just make sure the counters
we got back will neither be null nor be empty.
                
> Historyserver should provide counters for failed tasks
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-4693
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4693
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver, mrv2
>    Affects Versions: 2.0.3-alpha, 0.23.6
>            Reporter: Jason Lowe
>            Assignee: Xuan Gong
>              Labels: usability
>         Attachments: MAPREDUCE-4693.1.patch, MAPREDUCE-4693.2.patch
>
>
> Currently the historyserver is not providing counters for failed tasks, even though they
are available via the AM as long as the job is still running.  Those counters are lost when
the client needs to redirect to the historyserver after the job completes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message