hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Jeff.Schm...@shell.com>
Subject RE: Hadoop Discrete Event Simulator
Date Tue, 19 Jul 2011 12:17:35 GMT

You may want to check this out


-----Original Message-----
From: maneesh varshney [mailto:mvarshney@gmail.com] 
Sent: Monday, July 18, 2011 8:09 PM
To: common-user@hadoop.apache.org
Subject: Hadoop Discrete Event Simulator


Perhaps somebody can point out if there have been efforts to "simulate"
Hadoop clusters.

What I mean is a discrete event simulator that models the hosts and the
networks and run hadoop algorithms for some synthetic workload.
similar to network simulators (for example, ns2).

If such as tool is available, I was hoping to use it for:
a. Getting a general sense of how the HDFS and MapReduce algorithms
For example, if I were to store 1TB data over 100 nodes, how would the
blocks get distributed.
b. Use the simulation to optimize my configuration parameters. For
the relationship between performance and number of cluster node, or
of replicas, and so on.

The need for point b. above is to be able to study/analyze the
without (or before) actually running the algorithms on an actual

Thanks in advance,

PS: I apologize if this question has been asked earlier. I could not
seem to
locate the search feature in the mailing list archive.

View raw message