hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2652) MR-279: Cannot run multiple NMs on a single node
Date Tue, 30 Aug 2011 14:25:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093763#comment-13093763
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-2652:
----------------------------------------------------

bq. By changing the TaskAttemptEvent to a different class it exposed that I had not handled
the start container event consistently everywhere it was sent. I missed app master recovery,
local mode, and MRApp test mode. So I have updated them all.
Excellent! That did help after all :)

bq. Also the unit test changes you wanted are more stubbed out then I expected... I have tested
it manually and I can see how it would be nice to have a unit test to verify that, but that
would require some deep changes to MRApp to make that work.
I actually mentioned TestMRJobs or MiniMRYarnCluster in my comment. So you could set the port
number to 8081/8082 in MiniMRYarnCluster where shuffle server itself is set (MiniMRYarnCluster.java
+75). That should be enough - TestMRJobs, TestMRJobsWithHistoryService etc will automatically
test your code.


> MR-279: Cannot run multiple NMs on a single node 
> -------------------------------------------------
>
>                 Key: MAPREDUCE-2652
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2652
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>             Fix For: 0.23.0
>
>         Attachments: MR-2652-trunk-v3.patch, MR-2652-trunk-v4.patch, MR-2652-v1.txt,
MR-2652-v2.txt
>
>
> Currently in MR-279 the Auxiliary services, like ShuffleHandler, have no way to communicate
information back to the applications.  Because of this the Map Reduce Application Master has
hardcoded in a port of 8080 for shuffle.  This prevents the configuration "mapreduce.shuffle.port"
form ever being set to anything but 8080.  The code should be updated to allow this information
to be returned to the application master.  Also the data needs to be persisted to the task
log so that on restart the data is not lost.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message