hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Exponential performance decay - mystery solved
Date Thu, 21 Jan 2010 20:37:52 GMT
Some of this delay in generating block reports might be mitigated via
http://issues.apache.org/jira/browse/HDFS-854

thanks,
dhruba


On Thu, Jan 21, 2010 at 11:41 AM, <Zlatin.Balevsky@barclayscapital.com>wrote:

>
>
> Alright, the problem was caused by me setting the frequency of a block
> report to 30 seconds.  The idea behind that was to create more load on
> the Namenode, but I didn't notice that those block reports were taking
> increasing amounts of time to generate.  During that time, a lock was
> held which I'm guessing didn't allow the reporting datanode to perform
> its functions.
>
> On my hardware, with 100,000 blocks the report takes over 7 seconds.  So
> every datanode was unavailable for 7 out of every 30 seconds.  Changing
> the interval to a more reasonable value restored the insertion speed to
> linear.
>
> Apologies for creating this confusion, nevertheless it was a useful
> thing to learn.
>
> Regards,
> Zlatin
>
> -----Original Message-----
> From: Eli Collins [mailto:eli@cloudera.com]
> Sent: Thursday, January 21, 2010 2:02 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Exponential performance decay - possible lead
>
> >
> > The messages are of the following:
> >
> > 2010-01-18 14:51:25,694 WARN org.apache.hadoop.hdfs.StateChange:
> > BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request
> > received for blk_-5804440919363539694_1026 on ip.removed:port.removed
> > size 1024
>
> This is odd, you should't be getting this warning, I don't see it when
> running your benchmark on my cluster. Are there other relevant/warnings
> errors in the NN or DN logs?
>
> Thanks,
> Eli
> _______________________________________________
>
> This e-mail may contain information that is confidential, privileged or
> otherwise protected from disclosure. If you are not an intended recipient of
> this e-mail, do not duplicate or redistribute it by any means. Please delete
> it and any attachments and notify the sender that you have received it in
> error. Unless specifically indicated, this e-mail is not an offer to buy or
> sell or a solicitation to buy or sell any securities, investment products or
> other financial product or service, an official confirmation of any
> transaction, or an official statement of Barclays. Any views or opinions
> presented are solely those of the author and do not necessarily represent
> those of Barclays. This e-mail is subject to terms available at the
> following link: www.barcap.com/emaildisclaimer. By messaging with Barclays
> you consent to the foregoing.  Barclays Capital is the investment banking
> division of Barclays Bank PLC, a company registered in England (number
> 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.
>  This email may relate to or be sent from other members of the Barclays
> Group.
> _______________________________________________
>



-- 
Connect to me at http://www.facebook.com/dhruba

Mime
View raw message