hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Eagles (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
Date Tue, 11 Oct 2011 16:25:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125160#comment-13125160
] 

Jonathan Eagles commented on MAPREDUCE-2666:
--------------------------------------------

I went with the approach of logging this information into the jobhistory file. There was a
decision as to what event this information should be log to. I opted for the earliest possible
event the information is know, which is the TaskAttemptStarted event. 

Here is a sample event as logged in the job history file with the modifications.

{"type":"MAP_ATTEMPT_STARTED","event":{"org.apache.hadoop.mapreduce.jobhistory.TaskAttemptStarted":{"taskid":"task_1318349219359_0001_m_000000","taskType":"MAP","attemptId":"attempt_1318349219359_0001_m_000000_0","startTime":1318349
   308191,"trackerName":"machine.name.example.com","httpPort":9999,"shufflePort":8080}}}
                
> MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2666
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Robert Joseph Evans
>            Assignee: Jonathan Eagles
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on.  In the case
of an ApplicationMaster crash where it needs to be restarted that information is lost.  We
either need to re-query it from each of the NodeManagers or to persist it to the JobHistory
logs and retrieve it again.  The job history logs is probably the simpler solution.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message