hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Lederman <jon2...@gmail.com>
Subject Query
Date Sat, 04 Dec 2010 00:33:28 GMT
I am trying to run Hadoop on a multi-core architecture (e.g., 32 or more cores).  I am somewhat
unclear what the best configuration would be to maximize performance of Hadoop in this setting.

1) Since map/reduce jobs are i/o bound and all cores share the same i/o I am wondering whether
any significant  performance gains are possible.  

2) Will Hadoop leverage multiple cores automatically or do I need to set some options in the
configuration to do this?  How does Hadoop distribute map/reduce jobs in a multi-core setting?

3) The multi-core architecture allows each core to host a separate installation of Linux.
 So, I was wondering whether each core could operate as a Hadoop node.  The issue with this
I am concerned with is that all cores share the same i/o. Thus, I am not sure whether there
would be any performance benefits to setting up Hadoop in this way.

4) Any references or examples  that may discuss deploying and tuning Hadoop to leverage multi-core
architectures would be greatly appreciated.  Also, are there any imminent features coming
in future versions of Hadoop that might help in this respect?

Any help on these issues would be greatly appreciated. 
View raw message