hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xavier Stevens <Xavier.Stev...@fox.com>
Subject RE: Hadoop Write Performance
Date Wed, 18 Feb 2009 20:11:35 GMT

I was using, but I installed 0.18.3 a couple of days ago.  I
also separated out my secondarynamenode and jobtracker to another
machine.  In addition my network operations people had misconfigured
some switches which ended up being my bottleneck.

After all of that my writer and Hadoop is working great.


-----Original Message-----
From: Raghu Angadi [mailto:rangadi@yahoo-inc.com] 
Sent: Wednesday, February 18, 2009 11:49 AM
To: core-user@hadoop.apache.org
Subject: Re: Hadoop Write Performance

what is the hadoop version?

You could check log on a datanode around that time. You could post any
suspicious errors. For e.g. you can trace a particular block in client
and datanode logs.

Most likely it not a NameNode issue, but you can check NameNode log as


Xavier Stevens wrote:
> Does anyone have an expected or experienced write speed to HDFS 
> outside of Map/Reduce?  Any recommendations on properties to tweak in 
> hadoop-site.xml?
> Currently I have a multi-threaded writer where each thread is writing 
> to a different file.  But after a while I get this:
> java.io.IOException: Could not get block locations. Aborting...
>  at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(D
> FS
> Client.java:2081)
>  at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1300(DFSClient.
> ja
> va:1702)
>  at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl
> ie
> nt.java:1818)
> Which is perhaps indicating that the namenode is overwhelmed?
> Thanks,
> -Xavier

View raw message