hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radim Kolar <...@filez.com>
Subject Re: max 1 mapper per node
Date Thu, 10 May 2012 11:56:19 GMT

> We've been against these 'features' since it leads to very bad 
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which 
works good enough only for RAM. There are 2 other resources - CPU cores 
and network IO which needs to be considered.

We have job which is doing lot of network IO in mapper and its desirable 
to run mappers on different nodes even if reading blocks from HDFS will 
not be local.

Our second job is burning all CPU cores on machine while doing 
computations, its important for mappers not to land on same node.

Mime
View raw message