hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Zlatin.Balev...@barclayscapital.com>
Subject RE: Exponential performance decay when inserting large number of blocks
Date Fri, 15 Jan 2010 13:25:32 GMT
In the first few tests I created total of 24 files.  In the test with the last graph 88 files
before the break and 16 after.  The files were 

#!/bin/bash
HDFS_URI=$1
INSERT_NO=$2
for  i in $(seq 1 8)
do
 path/to/hadoop/bin/hadoop fs -put bigData hdfs://$HDFS_URI/bigData$INSERT_NO-$i > put$i.log
2>&1 &
done 

And I invoke it manually, increasing $2 each run.  
-----Original Message-----
From: Eli Collins [mailto:eli@cloudera.com] 
Sent: Thursday, January 14, 2010 9:13 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Exponential performance decay when inserting large number of blocks

On Thu, Jan 14, 2010 at 3:53 PM,  <Zlatin.Balevsky@barclayscapital.com> wrote:
> And the results after several hour wait are the same.  The cluster was 
> absolutely idle between the insertions.  Attached is a graph.

How many files are you creating?  Can you post the script you're using to drive the test?

Thanks,
Eli
_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected
from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or
redistribute it by any means. Please delete it and any attachments and notify the sender that
you have received it in error. Unless specifically indicated, this e-mail is not an offer
to buy or sell or a solicitation to buy or sell any securities, investment products or other
financial product or service, an official confirmation of any transaction, or an official
statement of Barclays. Any views or opinions presented are solely those of the author and
do not necessarily represent those of Barclays. This e-mail is subject to terms available
at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent
to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC,
a company registered in England (number 1026167) with its registered office at 1 Churchill
Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays
Group.
_______________________________________________

Mime
View raw message