On Thu, Jan 21, 2010 at 11:41 AM, = <Zlatin.Balevsky@barclay= scapital.com>=20 wrote:

Alright,=20 the problem was caused by me setting the frequency of a block
report t= o 30=20 seconds. The idea behind that was to create more load on
the=20 Namenode, but I didn't notice that those block reports were=20 taking
increasing amounts of time to generate. During that time,= a=20 lock was
held which I'm guessing didn't allow the reporting datanode t= o=20 perform
its functions.

On my hardware, with 100,000 blocks the= =20 report takes over 7 seconds. So
every datanode was unavailable f= or 7=20 out of every 30 seconds. Changing
the interval to a more reasona= ble=20 value restored the insertion speed to
linear.

Apologies for cre= ating=20 this confusion, nevertheless it was a useful
thing to=20 learn.

Regards,
Zlatin

-----Original Message-----
Fro= m:=20 Eli Collins [mailto:eli@cloudera.com]
Sent: Thursday,= =20 January 21, 2010 2:02 PM
To: hdfs-user@hadoop.apache.org
Subject:=20 Re: Exponential performance decay - possible lead

>
> The= =20 messages are of the following:
>
> 2010-01-18 14:51:25,694 WA= RN=20 org.apache.hadoop.hdfs.StateChange:
> BLOCK* NameSystem.addStoredBl= ock:=20 Redundant addStoredBlock request
> received for=20 blk_-5804440919363539694_1026 on ip.removed:port.removed
> size=20 1024

This is odd, you should't be getting this warning, I don't se= e it=20 when
running your benchmark on my cluster. Are there other=20 relevant/warnings
errors in the NN or DN=20 logs?

Thanks,
Eli
__________________________________________= _____

This=20 e-mail may contain information that is confidential, privileged or otherw= ise=20 protected from disclosure. If you are not an intended recipient of this=20 e-mail, do not duplicate or redistribute it by any means. Please delete i= t and=20 any attachments and notify the sender that you have received it in error.= =20 Unless specifically indicated, this e-mail is not an offer to buy or sell= or a=20 solicitation to buy or sell any securities, investment products or other= =20 financial product or service, an official confirmation of any transaction= , or=20 an official statement of Barclays. Any views or opinions presented are so= lely=20 those of the author and do not necessarily represent those of Barclays. T= his=20 e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Bar= clays=20 you consent to the foregoing. Barclays Capital is the investment ba= nking=20 division of Barclays Bank PLC, a company registered in England (number=20 1026167) with its registered office at 1 Churchill Place, London, E14 5HP= .=20 This email may relate to or be sent from other members of the Barcl= ays=20 Group.
_______________________________________________