hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alex kamil <alex.ka...@gmail.com>
Subject Re: Exponential performance decay when inserting large number of blocks
Date Thu, 14 Jan 2010 00:02:51 GMT
Zlotin, nevermind, just noticed that you're measuring the dfs insert time
1k block are way too small anyway

On Wed, Jan 13, 2010 at 6:08 PM, alex kamil <alex.kamil@gmail.com> wrote:

> Zlatin,
> i dont know what is the nature of the job that you are running but it looks
> like you are hitting an io bottleneck, try to multiple the size of input
> data and test with 3-5GB at least, by changing the block size in this case
> you can increase the number of maps and scale your app. (Round the block
> size to a multiple of 512)
> i attached some results from my experiments with KNN sclaing on 4 nodes
> cluster, it didnt scale linearly but it wasn't too bad, it sometimes depends
> on the algorithm, with other algos, like PCA/SVD the scaling was linear
> also note that sometimes the bottleneck in in Reduce step, i added a
> Combiner between mapper and reducer in my code and it affected scalability
> significantly.
>
> let me know if you have any questions
> Alex Kamil
> ak2834@columbia.edu
>
>
> On Wed, Jan 13, 2010 at 5:35 PM, Dhruba Borthakur <dhruba@gmail.com>wrote:
>
>> Another thing to observe is the rate of IO on the datanodes. Maybe u can
>> do a sar/iostat on the datanodes and see if the ddatanode devices show an
>> increase in activity while inserting the last lot of blocks. One posiblity
>> is that the OS cache on the datanodes cached most of the data from the first
>> few runs, but when more and more data started arriving on the datanode it
>> triggered more flushing of OS buffers. (on the datanode).
>>
>> thanks,
>> dhruba
>>
>>
>>
>> On Wed, Jan 13, 2010 at 2:18 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> Hey Zlatin,
>>>
>>> Thanks for the explanation and the additional data. I'm a bit busy today
>>> but will try to go through the data and reproduce the results later this
>>> week.
>>>
>>> -Todd
>>>
>>>
>>> On Wed, Jan 13, 2010 at 2:07 PM, <Zlatin.Balevsky@barclayscapital.com>wrote:
>>>
>>>>  Todd,
>>>>
>>>> I used a shell script that launched 8 instances of the bin/hadoop fs
>>>> -put utility.  After all 8 processes were done and I verified though the
web
>>>> ui that the files were inserted, I re-launched the script manually again.
>>>> That is why you'll notice that in the metrics there are two short periods
>>>> without any activity (I edited those out from the graph).  There were
>>>> occasional NotReplicatedYet exceptions in the logs of those processes, but
>>>> they were occurring at constant rate.
>>>>
>>>> I did not run a profiler, but that will eventually be the next step.
>>>> I'm attaching the metrics from the namenode and one of the datanodes from
>>>> the experiment with 4 datanodes.  They were recorded every 10 seconds.  Heap
>>>> size for all processes is 2GB, and while there was occasional CPU usage on
>>>> the Namenode it was never 100%.  (and there are plenty of cores).
>>>>
>>>> Ultimately the block size will be much larger than the default as the
>>>> total data will be in the 2^(well over 50) range.  With this test I am
>>>> trying to determine if there are any bottlenecks at the NameNode  component.
>>>>
>>>> Best Regards,
>>>> Zlatin Balevsky
>>>>
>>>>  ------------------------------
>>>> *From:* Todd Lipcon [mailto:todd@cloudera.com]
>>>> *Sent:* Wednesday, January 13, 2010 4:34 PM
>>>> *To:* hdfs-user@hadoop.apache.org
>>>> *Subject:* Re: Exponential performance decay when inserting large
>>>> number of blocks
>>>>
>>>> Also, if you have the program you used to do the insertions, and could
>>>> attach it, I'd be interested in trying to replicate this on a test cluster.
>>>> If you can't redistribute it, I can start from scratch, but would be easier
>>>> to run yours.
>>>>
>>>> Thanks
>>>> -Todd
>>>>
>>>> On Wed, Jan 13, 2010 at 1:31 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>>>
>>>>> Hi Zlatin,
>>>>>
>>>>> This is a very interesting test you've run, and certainly not expected
>>>>> results. I know of many clusters happily chugging along with millions
of
>>>>> blocks, so problems at 400K are very strange. By any chance were you
able to
>>>>> collect profiling information from the NameNode while running this test?
>>>>>
>>>>> That said, I hope you've set the block size to 1KB for the purpose of
>>>>> this test and not because you expect to run that in production. Recommended
>>>>> block sizes are at least 64MB and often 128MB or 256MB for larger clusters.
>>>>>
>>>>> Thanks
>>>>> -Todd
>>>>>
>>>>> On Wed, Jan 13, 2010 at 1:21 PM, <Zlatin.Balevsky@barclayscapital.com>wrote:
>>>>>
>>>>>> Greetings,
>>>>>>
>>>>>> I am testing how HDFS scales with very large number of blocks.  I
did
>>>>>> the following setup:
>>>>>>
>>>>>> Set the default blocks size to 1KB
>>>>>> Started 8 insert processes, each inserting a 16MB file
>>>>>> Repeated the insert 3 times, keeping the already inserted files in
>>>>>> HDFS
>>>>>> Repeated the entire experiment on one cluster with 4 and another
with
>>>>>> 11
>>>>>> identical datanodes (allocated through HOD)
>>>>>>
>>>>>> Results:
>>>>>> The first 128MB (2^18 blocks) insert finished in 5 minutes.  The
>>>>>> second
>>>>>> in 12 minutes.  The third didn't finish within 1 hour.  The 11-node
>>>>>> cluster was marginally faster.
>>>>>>
>>>>>> Throughout this I was storing all available metrics.  There were
no
>>>>>> signs of insufficient memory on any of the nodes; CPU usage and
>>>>>> garbage
>>>>>> collections were constant throughout.  If anyone is interested I
can
>>>>>> provide the recorded metrics.  I've attached a chart that looks
>>>>>> clearly
>>>>>> logarithmic.
>>>>>>
>>>>>> Can anyone please point to what could be the bottleneck here?  I'm
>>>>>> evaluating HDFS for usage scenarios requiring 2^(a lot more than
18)
>>>>>> blocks.
>>>>>>
>>>>>> Bes <<insertion_rate_4_and_11_datanodes.JPG>> t Regards,
>>>>>> Zlatin Balevsky
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> This e-mail may contain information that is confidential, privileged
>>>>>> or otherwise protected from disclosure. If you are not an intended
recipient
>>>>>> of this e-mail, do not duplicate or redistribute it by any means.
Please
>>>>>> delete it and any attachments and notify the sender that you have
received
>>>>>> it in error. Unless specifically indicated, this e-mail is not an
offer to
>>>>>> buy or sell or a solicitation to buy or sell any securities, investment
>>>>>> products or other financial product or service, an official confirmation
of
>>>>>> any transaction, or an official statement of Barclays. Any views
or opinions
>>>>>> presented are solely those of the author and do not necessarily represent
>>>>>> those of Barclays. This e-mail is subject to terms available at the
>>>>>> following link: www.barcap.com/emaildisclaimer. By messaging with
>>>>>> Barclays you consent to the foregoing.  Barclays Capital is the investment
>>>>>> banking division of Barclays Bank PLC, a company registered in England
>>>>>> (number 1026167) with its registered office at 1 Churchill Place,
London,
>>>>>> E14 5HP.  This email may relate to or be sent from other members
of the
>>>>>> Barclays Group.
>>>>>> _______________________________________________
>>>>>>
>>>>>
>>>>>
>>>>  _______________________________________________
>>>>
>>>>
>>>>
>>>> This e-mail may contain information that is confidential, privileged or
>>>> otherwise protected from disclosure. If you are not an intended recipient
of
>>>> this e-mail, do not duplicate or redistribute it by any means. Please delete
>>>> it and any attachments and notify the sender that you have received it in
>>>> error. Unless specifically indicated, this e-mail is not an offer to buy
or
>>>> sell or a solicitation to buy or sell any securities, investment products
or
>>>> other financial product or service, an official confirmation of any
>>>> transaction, or an official statement of Barclays. Any views or opinions
>>>> presented are solely those of the author and do not necessarily represent
>>>> those of Barclays. This e-mail is subject to terms available at the
>>>> following link: www.barcap.com/emaildisclaimer. By messaging with
>>>> Barclays you consent to the foregoing.  Barclays Capital is the
>>>> investment banking division of Barclays Bank PLC, a company registered in
>>>> England (number 1026167) with its registered office at 1 Churchill Place,
>>>> London, E14 5HP.  This email may relate to or be sent from other
>>>> members of the Barclays Group.**
>>>>
>>>> _______________________________________________
>>>>
>>>
>>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>
>

Mime
View raw message