hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Schulz <danielschulz2...@hotmail.com>
Subject Re: I/O time when reading from HDFS in Hadoop
Date Sat, 11 Jun 2016 16:52:25 GMT
Hello Alexandru,

So iff you are solely interested in the latencies, why not using the Linux' time command from
the shell. Just use the Hadoop CLI to get your file, try this from several nodes from various
racks for differing files from your cluster and build a Confidence Interval for the time it
took to retrieve each file from any node & rack.

Otherwise, a more holistic approach was to use this project: epaulson.github.io/HadoopInternals/benchmarks.html
Its Ohio State Infiniband benchmark contains latency information on sequential and random
writes on Read and Write operations and more.

Hope this helps…

Kind regards, Daniel.



Sent from my iPad
> On 11 Jun 2016, at 17:22, Alexandru Calin <alexandrucalin29@gmail.com> wrote:
> 
> Hello, 
> 
> I would like to measure the time taken for map and reduce when performing I/O (reading
from HDFS) in Hadoop. I am using Yarn. Hadoop 2.6.0. What are the options for that?
> 
> Thanks
Mime
View raw message