hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitry Pushkarev" <u...@stanford.edu>
Subject RE: namenode multitreaded
Date Fri, 12 Sep 2008 17:47:12 GMT
I have 15+ million small files I like to process and move around..Thus my
operations doesn't really include datanodes - they're idle when I for
example do FS operations (like sort a bunch of new files written by
tasktracker to appropriate folders). Now I tried to use HADOOP_OPTS=-server
and it seems to help a little, but still performance isn't great. 

Perhaps problem is in the way I play with files - it's perl script over
davf2 over WebDav which uses native API. 

Can anyone give an example of a jython or jruby file that'd recursively go
over a hdfs folder and move all files to a different folder? (My programming
skills are very modest..)

-----Original Message-----
From: Raghu Angadi [mailto:rangadi@yahoo-inc.com] 
Sent: Friday, September 12, 2008 9:41 AM
To: core-user@hadoop.apache.org
Subject: Re: namenode multitreaded

The core of namenode functionality happens in single thread because of a 
global lock, unfortunately. The other cpus would still be used to some 
extent by network IO and other threads. Usually we don't see just one 
cpu at 100% and nothing else on the other cpus.

What kind of load do you have?


Dmitry Pushkarev wrote:
> Hi.
> My namenode runs on a 8-core server with lots of RAM, but it only uses one
> core (100%).
> Is it possible to tell namenode to use all available cores?
> Thanks.

View raw message