Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "Dmitry Pushkarev" <umka@stanford.edu>
To: <core-user@hadoop.apache.org>
References: <000001c9148b$bbe386b0$33aa9410$@edu>
 <48CA9BA8.8030703@yahoo-inc.com>
In-Reply-To: <48CA9BA8.8030703@yahoo-inc.com>
Subject: RE: namenode multitreaded
Date: Fri, 12 Sep 2008 10:47:12 -0700
Message-ID: <000001c914ff$96630ea0$c3292be0$@edu>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AckU9n5Jid5aPwoWTUaypWx7tC4eoAAB8Uig
Content-Language: en-us

I have 15+ million small files I like to process and move around..Thus my
operations doesn't really include datanodes - they're idle when I for
example do FS operations (like sort a bunch of new files written by
tasktracker to appropriate folders). Now I tried to use HADOOP_OPTS=-server
and it seems to help a little, but still performance isn't great. 

Perhaps problem is in the way I play with files - it's perl script over
davf2 over WebDav which uses native API. 

Can anyone give an example of a jython or jruby file that'd recursively go
over a hdfs folder and move all files to a different folder? (My programming
skills are very modest..)


-----Original Message-----
From: Raghu Angadi [mailto:rangadi@yahoo-inc.com] 
Sent: Friday, September 12, 2008 9:41 AM
To: core-user@hadoop.apache.org
Subject: Re: namenode multitreaded


The core of namenode functionality happens in single thread because of a 
global lock, unfortunately. The other cpus would still be used to some 
extent by network IO and other threads. Usually we don't see just one 
cpu at 100% and nothing else on the other cpus.

What kind of load do you have?

Raghu.

Dmitry Pushkarev wrote:
> Hi.
> 
>  
> 
> My namenode runs on a 8-core server with lots of RAM, but it only uses one
> core (100%).
> 
> Is it possible to tell namenode to use all available cores?
> 
>  
> 
> Thanks.
> 
>