hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Shukla <sagar_shu...@persistent.co.in>
Subject RE: Write operation in TestDFSIO
Date Tue, 07 Feb 2012 11:46:55 GMT
Hi Sesha,
         I think, I still more details on network configuration, size of each file, etc to
confirm my response.

But at a high-level as the number of files / data increases above the threshold limit then
server / OS will need to spend more time in file-handling throttled by network speed which
will cause throughput to decrease after a limit of network is reached.

Also system might be heavily loaded to do required number of IO operations and hence you might
see a drop in performance.

Thanks,
Sagar

From: Sesha Kumar [mailto:sesha911@gmail.com]
Sent: Tuesday, February 07, 2012 3:36 PM
To: hdfs-user
Subject: Write operation in TestDFSIO

Hey guys,
       I performed TestDFSIO benchmark on my hadoop cluster (1namenode and 2 datanodes).
       The benchmark was run for different number of files constituting 200mb size.
       The following image is a graph of the throughput IO rate against number of files.
       I want to know why the write operation is varying this way. There is no pattern of
constant increase or decrease.[WriteOperation.png]

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed.
If you are not the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems Ltd. does not accept
any liability for virus infected mails.

Mime
View raw message