hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: A list of some HDFS benchmarks
Date Thu, 05 Sep 2013 11:24:17 GMT

FWIW the work on formally specify (in the Computer Science notion of
"formally") is in HADOOP-9361; the HCFS work being driven by redhat is more
about testing.

Some extra ideas on benchmarking
# something to assess performance of cross FS operations
# it'd be nice to have something that would let you experiment with
different hardware options in that NN
# the gridmix3 MapReduce benchmarks can collect job use stats to generate
synthetic workloads. Maybe we could derive something similar from NN
metrics, so that we could build up a better pool of operations on different
workloads (e.g. HBase, Hive + Tez) and apply them.
# there's work needed on scalability tests across filesystems; for the
'9361 tests I'm making them per-FS programmable for options like max #of
files in a directory test, max filesize etc -any additions there would be

On 4 September 2013 22:27, Erik Paulson <epaulson@unit1127.com> wrote:

> Hello all -
> As part of a side project, I've been interested in HDFS benchmarking,
> particularly of the Namenode. To get started, I tried to track down a
> number of different benchmarks and collect a few observations about each.
> I've put together a list here:
> http://epaulson.github.io/HadoopInternals/benchmarks.html
> The benchmarks I included were:
> NNBench and NNBenchWithoutMR
> S-Live
> LoadGenerator
> NNThroughputBenchmark
> TestEditLog
> MStress, from Quantcast
> Ohio State Microbenchmarks
> (I also wrote a bit about what else I'd like to see in a NN benchmark)
> I'd appreciate any corrections, feedback, and pointers to code that I
> missed!
> Thanks!
> -Erik

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message