hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Shukla <sagar_shu...@persistent.co.in>
Subject RE: response time increases lineraly for each thread!
Date Wed, 14 Apr 2010 04:22:45 GMT
Hi Gokul,
     Just to add to what Todd mentioned, in Hadoop - though you are uploading the file on
local disk but it would get distributed across multiple nodes maintaining the replication
factor and hence will generate network traffic. So this can affect performance as time goes
by.

Thanks,
Sagar

From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, April 14, 2010 2:04 AM
To: gokulm@huawei.com
Cc: hdfs-user@hadoop.apache.org; Sagar Shukla
Subject: Re: response time increases lineraly for each thread!

Hi Gokul,

This linear degradation makes sense. If you're saturating your outgoing network bandwidth
from the node that's doing the writing, then doubling the number of writers will decrease
the amount of bandwidth available to each writer by a factor of two. So of course each writer
will take twice as long if there are twice as many writers.

This is true regardless of whether you are disk bound or network bound on the local node -
you have a finite resource (throughput) which you're splitting among more clients.

If you run the concurrent writers from different nodes you'll probably see that the write
time stays constant until you start to saturate bandwidth on all of the nodes.

Does this make sense?
-Todd
On Tue, Apr 13, 2010 at 2:15 AM, Gokulakannan M <gokulm@huawei.com<mailto:gokulm@huawei.com>>
wrote:
Todd,

            I am benchmarking the performance for the write scenario(Not only for single user
but for concurrent users). I've implemented a simple FileWrite() method that takes namenode
address and source file and write into hdfs. I'm measuring the response time by just getting
the time diff before and after calling the FileWrite() method (by using System.currentMillliSeconds()).I've
created several threads for this FileWrite() method and measure the performance. Is anything
wrong with the test code?


            No I didn't configure dfs.data.dir  to all volumes but the current one. So the
issue might be due to disk latency? The total space available is 750GB(seen from name UI).

 Thanks,
  Gokul

  [cid:image001.gif@01CADBB8.398A97D0]
________________________________
From: Todd Lipcon [mailto:todd@cloudera.com<mailto:todd@cloudera.com>]
Sent: Tuesday, April 13, 2010 10:54 AM
To: gokulm@huawei.com<mailto:gokulm@huawei.com>; hdfs-user@hadoop.apache.org<mailto:hdfs-user@hadoop.apache.org>
Cc: sagar_shukla@persistent.co.in<mailto:sagar_shukla@persistent.co.in>

Subject: Re: response time increases lineraly for each thread!

Hi Gokul,

Would it be possible to post your benchmark code somewhere? I'm not clear on what exact thing
you're measuring - IOUtils.copyBytes is a pretty general method.

Also, what disk hardware are you using? Have you configured dfs.data.dir to point to all local
drives on each DN? Could very well be a factor of disk seeks limiting you.
-Todd


On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <gokulm@huawei.com<mailto:gokulm@huawei.com>>
wrote:
@Todd,

            I am using 1 namenode and 10 datanodes. Replication factor is 3.

            These are the details about my test suite. It uses the simple write code that
uses IOUtils.copyBytes() and calculates the response time for each write for a given number
of users(threads).

            From namenode, I am running the test suite which calculates the response time
for writing a file for several concurrent users(1 ,  4  , 9 ,  20 ). Whatever the file size
may be(I tried with 100 MB , 1 GB and 5 GB files), I find that the response time increases
linearly.

            These are the results for a 100 MB file.


No of Threads

Response time(milli sec)

1

2199





4

2763



2774



34766



64488





9

3620



4897



5018



9991



65501



124156



183639



243631



314233





20

6702



6784



6985



8987



9202



9271



9925



10752



68237



70878



73469



75261



77507



82061



129942



137098



141822



194199



269353



322328



            You can see the write time increases for each thread.

@ Sagar:

            I think network is not the issue here. This is a private network with high bandwidth.
Moreover I tried the running the same test suite with several configurations(pseudo dist mode,
1 NN 1 DN Cluster , 1 NN 3 DN cluster) in my usual network(typically with low bandwidth) and
obtained the results like above :(


 Thanks,
  Gokul

  [cid:image001.gif@01CADBB8.398A97D0]

________________________________
From: Todd Lipcon [mailto:todd@cloudera.com<mailto:todd@cloudera.com>]
Sent: Monday, April 12, 2010 8:18 PM
To: hdfs-user@hadoop.apache.org<mailto:hdfs-user@hadoop.apache.org>; gokulm@huawei.com<mailto:gokulm@huawei.com>
Subject: Re: response time increases lineraly for each thread!

Can you be more specific about your benchmark? You're almost certainly being limited by either
outgoing network from your single node, or by disk throughput on that node (writes will go
local in addition to remote replicas if you are running a DN on the same host)

-Todd
On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <gokulm@huawei.com<mailto:gokulm@huawei.com>>
wrote:

Hi,

            When I tested the performance of a 11 node hadoop cluster(private nw) for the
write scenario with several threads, I noticed that the time for each write increases linearly(what
ever the file size may be).

            Actually I'm running these threads as a single process in the system where namenode
is running. I'm getting inconsistent results when I run this suite each time. The results
are fluctuating. What might be the bottlenecks?

            I think running this in parallel by spawning multiple shells will give consistent
result. But running these in the same system will give exact results??

            Any ways to measure the performance correctly?

            Appreciated any help in this.

 Thanks,
  Gokul





--
Todd Lipcon
Software Engineer, Cloudera



--
Todd Lipcon
Software Engineer, Cloudera



--
Todd Lipcon
Software Engineer, Cloudera

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed.
If you are not the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems Ltd. does not accept
any liability for virus infected mails.

Mime
View raw message