hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2425) Distributed simulator for stressing JobTracker and NameNode
Date Fri, 08 Apr 2011 05:43:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017296#comment-13017296
] 

Amar Kamat commented on MAPREDUCE-2425:
---------------------------------------

Min,
Thanks for the quick reply. I guess testing (stress/functional) JT is also the primary goal
of Mumak. Hence I see this as an enhancement (maybe rewrite) of Mumak. So the goal is to support
existing functionality of Mumak and add new features to support the use cases that you are
interested in. 

bq. Mumak uses a simulated JT.. 
I guess Mumak only wanted to work with certain features of JT and hence the design. I believe
we can enhance Mumak to instantiate the "real" JT if needed.

bq. I should uses new MR API before merging into mumak.
Yes.

For now, can you quickly point out the major highlights of the simulator that you are planning
to contribute? It would be nice to also compare/contrast it with Mumak.


> Distributed simulator for stressing JobTracker and NameNode
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-2425
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2425
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: benchmarks
>            Reporter: Min Zhou
>              Labels: benchmark, hadoop
>             Fix For: 0.22.0
>
>         Attachments: .jpg, screenshot-1.jpg
>
>
> Hadoop need a tool for stressing JobTracker and NameNode. Mumak introduced a simulated
JobTracker, whose behavior doesn't exactly like that of the real JobTracker. Even more, mumak
can't simulate a large cluster with quite a lot of jobs run on it. On the other hand, Gridmix
v3 need hundreds of physical nodes to replay job stories. 
> You can think this tool a complementation of mumak and gridmix v3. We successfully used
this tool to simulate a 12000 nodes cluster through 4 real machines. 
> I've talk to Hong Tang and Scott Chen offline, they suggested me contributing this tool
to the hadoop community.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message