hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Profiling HDFS
Date Wed, 29 Dec 2010 05:47:54 GMT
how frequently do you open new files to write? Or do you continue to write
to the same file(s) for the entire duration of the test? what is ur block
size? can you pl elaborate on your test workload?

On Tue, Dec 28, 2010 at 9:45 PM, Rajat Goel <rajatgoel06@gmail.com> wrote:

> Hi,
> I want to measure read/write rates to HDFS under various conditions such as
> under heavy load or one data node goes down etc? Is there some profiler
> already available for such purpose?
> I am pushing data at high rate to HDFS, reads are also happening in
> parallel and I suddenly reboot one datanode. I observe that I am no longer
> able to write to HDFS (from live datanodes) at the same higher rate. This
> happens for few minutes (around 30 mins), after which things go back to
> normal again. I want to find out why HDFS becomes slow, what is the main
> contributor of this latency and can I improve this behavior by changing some
> configuration parameters.
> Thanks & Regards,
> Rajat

Connect to me at http://www.facebook.com/dhruba

View raw message