hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj K <devara...@huawei.com>
Subject RE: RE: Can I use the cores of each CPU to be the datanodes instead of CPU?
Date Mon, 08 Aug 2011 09:37:51 GMT
Hi Jun Tan,

 

   As Subroto pointed Hadoop doesn’t  depend on the number of cores but we
can tune the parameters based on the no of cores and type of job we are
running for better performance. 

 

We can start multiple data nodes or task trackers irrespective of single
core or multi core by doing some changes in the script and configuration
files. 

 

You can have a look into this, for what all changes need to do for starting
multiple data nodes in a single machine.

 

http://www.mail-archive.com/hdfs-user@hadoop.apache.org/msg01353.html

Devaraj K 

  _____  

From: 谭军 [mailto:tanjun_2525@163.com] 
Sent: Monday, August 08, 2011 2:49 PM
To: mapreduce-user@hadoop.apache.org; subroto.sanyal@huawei.com
Subject: Re:RE: Can I use the cores of each CPU to be the datanodes instead
of CPU?

 

Hi Subroto,

I'm sorry for my poor English.

 

Are you thinking about CPU core to Hadoop process mapping?

Maybe this is the issue.

 

2 computers with 2 CPUs.

Each CPU has 2 cores.

Now I have 2 physical datanodes.

Can I get 4 physical datanodes?

I don't know wether I make my point clear?


 

--

Regards!

Jun Tan


在 2011-08-08 17:05:31,"Subroto Sanyal" <subrotosanyal@huawei.com> 写道:



 

Hi Jun,

 

Sorry, I didn’t get your question properly…..  :-(

 

Are you thinking in terms of Virtual Machines running on same physical
server?

Are you thinking about CPU core to Hadoop process mapping?

 

As per my understanding, Hadoop doesn’t have intelligence of different
cores (ideally application should be independent of hardware details)

 

It is possible to run multiple instance of DataNode in same physical server,
provide the configurations are different for each DataNode process.

 

Regards, 
Subroto Sanyal

  _____  

From: 谭军 [mailto:tanjun_2525@163.com] 
Sent: Monday, August 08, 2011 2:16 PM
To: mapreduce
Subject: Can I use the cores of each CPU to be the datanodes instead of CPU?

 

Hi,

I have 2 computers each of which is double-core CPU.

The 2 computers are datanodes and another computer is the namenode.

Now, I want to increase the number of datanode but no more computers
available.

Can I use the 4 cores of the 2 computers to be 4 datanodes?




--

Best regards!

Jun Tan













Mime
View raw message