hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maneesh varshney <mvarsh...@gmail.com>
Subject Hadoop Discrete Event Simulator
Date Tue, 19 Jul 2011 01:08:58 GMT

Perhaps somebody can point out if there have been efforts to "simulate"
Hadoop clusters.

What I mean is a discrete event simulator that models the hosts and the
networks and run hadoop algorithms for some synthetic workload. Something
similar to network simulators (for example, ns2).

If such as tool is available, I was hoping to use it for:
a. Getting a general sense of how the HDFS and MapReduce algorithms work.
For example, if I were to store 1TB data over 100 nodes, how would the
blocks get distributed.
b. Use the simulation to optimize my configuration parameters. For example,
the relationship between performance and number of cluster node, or number
of replicas, and so on.

The need for point b. above is to be able to study/analyze the performance
without (or before) actually running the algorithms on an actual cluster.

Thanks in advance,

PS: I apologize if this question has been asked earlier. I could not seem to
locate the search feature in the mailing list archive.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message