hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Tang (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-728) Mumak: Map-Reduce Simulator
Date Thu, 17 Sep 2009 20:22:57 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hong Tang updated MAPREDUCE-728:
--------------------------------

    Affects Version/s: 0.21.0
               Status: Patch Available  (was: Open)

To review/commit the patch, follow the steps below:
- apply patch http://issues.apache.org/jira/secure/attachment/12419875/mapred-995-v1.patch
- apply patch mapreduce-728-20090917.patch
- download the two json.gz files and store them under src/contrib/mumak/src/test/data
- "ant jar tools"
- "cd src/contrib/mumak && ant test"


> Mumak: Map-Reduce Simulator
> ---------------------------
>
>                 Key: MAPREDUCE-728
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-728
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: 19-jobs.topology.json.gz, 19-jobs.trace.json.gz, mapreduce-728-20090917.patch,
mumak.png
>
>
> h3. Vision:
> We want to build a Simulator to simulate large-scale Hadoop clusters, applications and
workloads. This would be invaluable in furthering Hadoop by providing a tool for researchers
and developers to prototype features (e.g. pluggable block-placement for HDFS, Map-Reduce
schedulers etc.) and predict their behaviour and performance with reasonable amount of confidence,
there-by aiding rapid innovation.
> ----
> h3. First Cut: Simulator for the Map-Reduce Scheduler
> The Map-Reduce Scheduler is a fertile area of interest with at least four schedulers,
each with their own set of features, currently in existence: Default Scheduler, Capacity Scheduler,
Fairshare Scheduler & Priority Scheduler.
> Each scheduler's scheduling decisions are driven by many factors, such as fairness, capacity
guarantee, resource availability, data-locality etc.
> Given that, it is non-trivial to accurately choose a single scheduler or even a set of
desired features to predict the right scheduler (or features) for a given workload. Hence
a simulator which can predict how well a particular scheduler works for some specific workload
by quickly iterating over schedulers and/or scheduler features would be quite useful.
> So, the first cut is to implement a simulator for the Map-Reduce scheduler which take
as input a job trace derived from production workload and a cluster definition, and simulates
the execution of the jobs in as defined in the trace in this virtual cluster. As output, the
detailed job execution trace (recorded in relation to virtual simulated time) could then be
analyzed to understand various traits of individual schedulers (individual jobs turn around
time, throughput, faireness, capacity guarantee, etc). To support this, we would need a simulator
which could accurately model the conditions of the actual system which would affect a schedulers
decisions. These include very large-scale clusters (thousands of nodes), the detailed characteristics
of the workload thrown at the clusters, job or task failures, data locality, and cluster hardware
(cpu, memory, disk i/o, network i/o, network topology) etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message