hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Zlatin.Balev...@barclayscapital.com>
Subject RE: Exponential performance decay when inserting large number of blocks
Date Thu, 14 Jan 2010 01:12:27 GMT
Alex, Dhruba
 
I repeated the experiment increasing the block size to 32k.  Still doing
8 inserts in parallel, file size now is 512 MB; 11 datanodes.  I was
also running iostat on one of the datanodes.  Did not notice anything
that would explain an exponential slowdown.  There was more activity
while the inserts were active but far from the limits of the disk
system.
 
There is definitely some improvement - I was able to finish 3 rounds of
inserts for total of 12GB.  However, the shape is still logarithmic.
Unfortunately, I am constrained with disk space and cannot test a block
size that is even close to the real world sizes.  
 
Attached are the collected metrics and a graph comparing the three
inserts.  As before, you'll notice gaps in the metrics between runs of
the insert script which have been edited out of the graph.
 
Best Regards,
Zlatin Balevsky
 
 


________________________________

From: alex kamil [mailto:alex.kamil@gmail.com] 
Sent: Wednesday, January 13, 2010 7:03 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Exponential performance decay when inserting large number
of blocks


Zlotin, nevermind, just noticed that you're measuring the dfs insert
time 
1k block are way too small anyway


On Wed, Jan 13, 2010 at 6:08 PM, alex kamil <alex.kamil@gmail.com>
wrote:


	Zlatin, 
	i dont know what is the nature of the job that you are running
but it looks like you are hitting an io bottleneck, try to multiple the
size of input data and test with 3-5GB at least, by changing the block
size in this case you can increase the number of maps and scale your
app. (Round the block size to a multiple of 512)
	i attached some results from my experiments with KNN sclaing on
4 nodes cluster, it didnt scale linearly but it wasn't too bad, it
sometimes depends on the algorithm, with other algos, like PCA/SVD the
scaling was linear  
	also note that sometimes the bottleneck in in Reduce step, i
added a Combiner between mapper and reducer in my code and it affected
scalability significantly.
	
	let me know if you have any questions
	Alex Kamil
	ak2834@columbia.edu 


	On Wed, Jan 13, 2010 at 5:35 PM, Dhruba Borthakur <
dhruba@gmail.com> wrote:
	

		Another thing to observe is the rate of IO on the
datanodes. Maybe u can do a sar/iostat on the datanodes and see if the
ddatanode devices show an increase in activity while inserting the last
lot of blocks. One posiblity is that the OS cache on the datanodes
cached most of the data from the first few runs, but when more and more
data started arriving on the datanode it triggered more flushing of OS
buffers. (on the datanode).
		
		thanks,
		dhruba 



		On Wed, Jan 13, 2010 at 2:18 PM, Todd Lipcon <
todd@cloudera.com> wrote:
		

			Hey Zlatin, 

			Thanks for the explanation and the additional
data. I'm a bit busy today but will try to go through the data and
reproduce the results later this week.

			-Todd 


			On Wed, Jan 13, 2010 at 2:07 PM, <
Zlatin.Balevsky@barclayscapital.com> wrote:
			

				Todd,
				 
				I used a shell script that launched 8
instances of the bin/hadoop fs -put utility.  After all 8 processes were
done and I verified though the web ui that the files were inserted, I
re-launched the script manually again.  That is why you'll notice that
in the metrics there are two short periods without any activity (I
edited those out from the graph).  There were occasional
NotReplicatedYet exceptions in the logs of those processes, but they
were occurring at constant rate.
				 
				I did not run a profiler, but that will
eventually be the next step.  I'm attaching the metrics from the
namenode and one of the datanodes from the experiment with 4 datanodes.
They were recorded every 10 seconds.  Heap size for all processes is
2GB, and while there was occasional CPU usage on the Namenode it was
never 100%.  (and there are plenty of cores).
				 
				Ultimately the block size will be much
larger than the default as the total data will be in the 2^(well over
50) range.  With this test I am trying to determine if there are any
bottlenecks at the NameNode  component.
				 
				Best Regards,
				Zlatin Balevsky
				 
________________________________

				From: Todd Lipcon [mailto:
todd@cloudera.com] 
				Sent: Wednesday, January 13, 2010 4:34
PM
				To: hdfs-user@hadoop.apache.org
				Subject: Re: Exponential performance
decay when inserting large number of blocks
				
				
				Also, if you have the program you used
to do the insertions, and could attach it, I'd be interested in trying
to replicate this on a test cluster. If you can't redistribute it, I can
start from scratch, but would be easier to run yours. 

				Thanks
				-Todd
				
				
				On Wed, Jan 13, 2010 at 1:31 PM, Todd
Lipcon <todd@cloudera.com> wrote:
				

				Hi Zlatin, 

				This is a very interesting test you've
run, and certainly not expected results. I know of many clusters happily
chugging along with millions of blocks, so problems at 400K are very
strange. By any chance were you able to collect profiling information
from the NameNode while running this test?

				That said, I hope you've set the block
size to 1KB for the purpose of this test and not because you expect to
run that in production. Recommended block sizes are at least 64MB and
often 128MB or 256MB for larger clusters.

				Thanks
				-Todd

				On Wed, Jan 13, 2010 at 1:21 PM, <
Zlatin.Balevsky@barclayscapital.com> wrote:
				

				Greetings,
				
				I am testing how HDFS scales with very
large number of blocks.  I did
				the following setup:
				
				Set the default blocks size to 1KB
				Started 8 insert processes, each
inserting a 16MB file
				Repeated the insert 3 times, keeping the
already inserted files in HDFS
				Repeated the entire experiment on one
cluster with 4 and another with 11
				identical datanodes (allocated through
HOD)
				
				Results:
				The first 128MB (2^18 blocks) insert
finished in 5 minutes.  The second
				in 12 minutes.  The third didn't finish
within 1 hour.  The 11-node
				cluster was marginally faster.
				
				Throughout this I was storing all
available metrics.  There were no
				signs of insufficient memory on any of
the nodes; CPU usage and garbage
				collections were constant throughout.
If anyone is interested I can
				provide the recorded metrics.  I've
attached a chart that looks clearly
				logarithmic.
				
				Can anyone please point to what could be
the bottleneck here?  I'm
				evaluating HDFS for usage scenarios
requiring 2^(a lot more than 18)
				blocks.
				
				Bes
<<insertion_rate_4_and_11_datanodes.JPG>> t Regards,
				Zlatin Balevsky
				
	
_______________________________________________
				
				This e-mail may contain information that
is confidential, privileged or otherwise protected from disclosure. If
you are not an intended recipient of this e-mail, do not duplicate or
redistribute it by any means. Please delete it and any attachments and
notify the sender that you have received it in error. Unless
specifically indicated, this e-mail is not an offer to buy or sell or a
solicitation to buy or sell any securities, investment products or other
financial product or service, an official confirmation of any
transaction, or an official statement of Barclays. Any views or opinions
presented are solely those of the author and do not necessarily
represent those of Barclays. This e-mail is subject to terms available
at the following link: www.barcap.com/emaildisclaimer. By messaging with
Barclays you consent to the foregoing.  Barclays Capital is the
investment banking division of Barclays Bank PLC, a company registered
in England (number 1026167) with its registered office at 1 Churchill
Place, London, E14 5HP.  This email may relate to or be sent from other
members of the Barclays Group.
	
_______________________________________________
				



	
_______________________________________________

				 

				This e-mail may contain information that
is confidential, privileged or otherwise protected from disclosure. If
you are not an intended recipient of this e-mail, do not duplicate or
redistribute it by any means. Please delete it and any attachments and
notify the sender that you have received it in error. Unless
specifically indicated, this e-mail is not an offer to buy or sell or a
solicitation to buy or sell any securities, investment products or other
financial product or service, an official confirmation of any
transaction, or an official statement of Barclays. Any views or opinions
presented are solely those of the author and do not necessarily
represent those of Barclays. This e-mail is subject to terms available
at the following link: www.barcap.com/emaildisclaimer. By messaging with
Barclays you consent to the foregoing.  Barclays Capital is the
investment banking division of Barclays Bank PLC, a company registered
in England (number 1026167) with its registered office at 1 Churchill
Place, London, E14 5HP.  This email may relate to or be sent from other
members of the Barclays Group.

	
_______________________________________________





		-- 
		Connect to me at http://www.facebook.com/dhruba
		




_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected
from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or
redistribute it by any means. Please delete it and any attachments and notify the sender that
you have received it in error. Unless specifically indicated, this e-mail is not an offer
to buy or sell or a solicitation to buy or sell any securities, investment products or other
financial product or service, an official confirmation of any transaction, or an official
statement of Barclays. Any views or opinions presented are solely those of the author and
do not necessarily represent those of Barclays. This e-mail is subject to terms available
at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent
to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC,
a company registered in England (number 1026167) with its registered office at 1 Churchill
Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays
Group.
_______________________________________________

Mime
View raw message