hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Kubes <nutch-...@dragonflymc.com>
Subject Re: Multiple tasktrackers per node
Date Wed, 24 May 2006 14:31:05 GMT
Using Java 5 will allow the threads of various tasks to take advantage 
of multiple processors.  Just make sure you set you map tasks property 
to a multiple of the number of processors total.  We are running 
multi-core machines and are seeing good utilization across all cores 
this way.

Dennis



Gianlorenzo Thione wrote:
> Hello everybody,
>
> I'll ask my first question on this forum and hopefully start building 
> more and more understanding of hadoop so that we can eventually 
> contribute actively. In the meanwhile, I have a simple 
> issue/question/suggestion....
>
> I have many multi-core, multi-processor nodes in my cluster and I'd 
> like to be able to run several tasktrackers and datanode per physical 
> machine. I am modifying the startup scripts so that a number of worker 
> JVMs can be started on each node, maxed out at the number of CPUs seen 
> by the kernel.
>
> Since our map jobs are highly CPU intensive it makes sense to run 
> parallel jobs on each node, maximizing the CPU utilization.
>
> Is that something that would make sense to roll back in the scripts 
> for hadoop as well? Anybody else running on multi processor 
> architectures?
>
> Lorenzo Thione
> Powerset, Inc.
>

Mime
View raw message