hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gokulakannan M <gok...@huawei.com>
Subject RE: response time increases lineraly for each thread!
Date Wed, 14 Apr 2010 03:11:41 GMT
Hi Todd,

 

            This really makes sense. Thanks :-)

            

 Thanks,

  Gokul

 

  

 

  _____  

From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Wednesday, April 14, 2010 2:04 AM
To: gokulm@huawei.com
Cc: hdfs-user@hadoop.apache.org; sagar_shukla@persistent.co.in
Subject: Re: response time increases lineraly for each thread!

 

Hi Gokul,

 

This linear degradation makes sense. If you're saturating your outgoing
network bandwidth from the node that's doing the writing, then doubling the
number of writers will decrease the amount of bandwidth available to each
writer by a factor of two. So of course each writer will take twice as long
if there are twice as many writers.

 

This is true regardless of whether you are disk bound or network bound on
the local node - you have a finite resource (throughput) which you're
splitting among more clients.

 

If you run the concurrent writers from different nodes you'll probably see
that the write time stays constant until you start to saturate bandwidth on
all of the nodes.

 

Does this make sense?

-Todd

On Tue, Apr 13, 2010 at 2:15 AM, Gokulakannan M <gokulm@huawei.com> wrote:

Todd,

 

            I am benchmarking the performance for the write scenario(Not
only for single user but for concurrent users). I've implemented a simple
FileWrite() method that takes namenode address and source file and write
into hdfs. I'm measuring the response time by just getting the time diff
before and after calling the FileWrite() method (by using
System.currentMillliSeconds()).I've created several threads for this
FileWrite() method and measure the performance. Is anything wrong with the
test code?

 

 

            No I didn't configure dfs.data.dir  to all volumes but the
current one. So the issue might be due to disk latency? The total space
available is 750GB(seen from name UI).  

            

 Thanks,

  Gokul

 

  

  _____  

From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Tuesday, April 13, 2010 10:54 AM
To: gokulm@huawei.com; hdfs-user@hadoop.apache.org
Cc: sagar_shukla@persistent.co.in


Subject: Re: response time increases lineraly for each thread!

 

Hi Gokul,

 

Would it be possible to post your benchmark code somewhere? I'm not clear on
what exact thing you're measuring - IOUtils.copyBytes is a pretty general
method.

 

Also, what disk hardware are you using? Have you configured dfs.data.dir to
point to all local drives on each DN? Could very well be a factor of disk
seeks limiting you.

-Todd

 

 

On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <gokulm@huawei.com> wrote:

@Todd,

 

            I am using 1 namenode and 10 datanodes. Replication factor is 3.

 

            These are the details about my test suite. It uses the simple
write code that uses IOUtils.copyBytes() and calculates the response time
for each write for a given number of users(threads). 

 

            From namenode, I am running the test suite which calculates the
response time for writing a file for several concurrent users(1 ,  4  , 9 ,
20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB
files), I find that the response time increases linearly. 

 

            These are the results for a 100 MB file.

 

            


No of Threads

Response time(milli sec)


1

2199


 

 


4

2763


 

2774


 

34766


 

64488


 

 


9

3620


 

4897


 

5018


 

9991


 

65501


 

124156


 

183639


 

243631


 

314233


 

 


20

6702


 

6784


 

6985


 

8987


 

9202


 

9271


 

9925


 

10752


 

68237


 

70878


 

73469


 

75261


 

77507


 

82061


 

129942


 

137098


 

141822


 

194199


 

269353


 

322328

 

 

            You can see the write time increases for each thread.

 

@ Sagar:

 

            I think network is not the issue here. This is a private network
with high bandwidth. Moreover I tried the running the same test suite with
several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN 3 DN
cluster) in my usual network(typically with low bandwidth) and obtained the
results like above :(

            

 

 Thanks,

  Gokul

 

  

 

  _____  

From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Monday, April 12, 2010 8:18 PM
To: hdfs-user@hadoop.apache.org; gokulm@huawei.com
Subject: Re: response time increases lineraly for each thread!

 

Can you be more specific about your benchmark? You're almost certainly being
limited by either outgoing network from your single node, or by disk
throughput on that node (writes will go local in addition to remote replicas
if you are running a DN on the same host)

 

-Todd

On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <gokulm@huawei.com> wrote:

 

Hi,

 

            When I tested the performance of a 11 node hadoop
cluster(private nw) for the write scenario with several threads, I noticed
that the time for each write increases linearly(what ever the file size may
be).

 

            Actually I'm running these threads as a single process in the
system where namenode is running. I'm getting inconsistent results when I
run this suite each time. The results are fluctuating. What might be the
bottlenecks? 

 

            I think running this in parallel by spawning multiple shells
will give consistent result. But running these in the same system will give
exact results??

 

            Any ways to measure the performance correctly?

 

            Appreciated any help in this.

 

 Thanks,

  Gokul

 

 




-- 
Todd Lipcon
Software Engineer, Cloudera




-- 
Todd Lipcon
Software Engineer, Cloudera




-- 
Todd Lipcon
Software Engineer, Cloudera


Mime
View raw message