hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Zlatin.Balev...@barclayscapital.com>
Subject RE: Exponential performance decay - mystery solved
Date Thu, 21 Jan 2010 19:41:13 GMT

Alright, the problem was caused by me setting the frequency of a block
report to 30 seconds.  The idea behind that was to create more load on
the Namenode, but I didn't notice that those block reports were taking
increasing amounts of time to generate.  During that time, a lock was
held which I'm guessing didn't allow the reporting datanode to perform
its functions.

On my hardware, with 100,000 blocks the report takes over 7 seconds.  So
every datanode was unavailable for 7 out of every 30 seconds.  Changing
the interval to a more reasonable value restored the insertion speed to

Apologies for creating this confusion, nevertheless it was a useful
thing to learn.


-----Original Message-----
From: Eli Collins [mailto:eli@cloudera.com] 
Sent: Thursday, January 21, 2010 2:02 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Exponential performance decay - possible lead

> The messages are of the following:
> 2010-01-18 14:51:25,694 WARN org.apache.hadoop.hdfs.StateChange: 
> BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request 
> received for blk_-5804440919363539694_1026 on ip.removed:port.removed 
> size 1024

This is odd, you should't be getting this warning, I don't see it when
running your benchmark on my cluster. Are there other relevant/warnings
errors in the NN or DN logs?


This e-mail may contain information that is confidential, privileged or otherwise protected
from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or
redistribute it by any means. Please delete it and any attachments and notify the sender that
you have received it in error. Unless specifically indicated, this e-mail is not an offer
to buy or sell or a solicitation to buy or sell any securities, investment products or other
financial product or service, an official confirmation of any transaction, or an official
statement of Barclays. Any views or opinions presented are solely those of the author and
do not necessarily represent those of Barclays. This e-mail is subject to terms available
at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent
to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC,
a company registered in England (number 1026167) with its registered office at 1 Churchill
Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays

View raw message