hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Foley <ma...@yahoo-inc.com>
Subject Re: where is example of the configuration about multi nodes on one machine?
Date Tue, 30 Nov 2010 18:00:01 GMT
Here is a "recipe" for how to run multiple datanodes on a single server, posted to this list
on Sept. 15:
	http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201009.mbox/%3C8A898C33-DC4E-418C-ADC0-5689D434B05E@yahoo-inc.com%3E

If you're having trouble getting multiple cores utilized,  this should help, assuming the
computation is well expressed in Map/Reduce.  
There is no point in running VMs in your situation, as it just adds overhead without changing
the amount of physical processor power 
available.

--Matt

On Nov 30, 2010, at 1:21 AM, beneo_7 wrote:

> If you want to just use one machine, why do you want to use hadoop? Hadoop's
> power lies in distributed computing. That being said, it is possible to use
> hadoop on a single machine by using the pseudo-distributed mode (Read
> http://hadoop.apache.org/common/docs/current/single_node_setup.html and
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/).
> If you are using just one machine, at least ensure that your machine has
> lots of cores (8 core/16 cores would be great) to get benefit out of hadoop.
> 
> I am not sure, but using virtual machines won't be helpful here as a virtual
> machine is just an abstraction and not real hardware.


thanks very much, i use the hadoop because the apache Mahout project need it for clustering.

my machine is only one and powerful, 16cores and 32g mem, but i have only one, so i need configuration
about multi nodes on one machine.

i have used the "pseudo-distributed mode", however, the project always used 1 core,  the cpu
freq always 100% ~ 103%, the time the execution is 4 hours, it's too slow.

i can not change the mahout project source code, the trunk always update sometime, it's difficulty
to solute confliction.


is there any way to conf some slaves in one machine?

At 2010-11-30 17:07:49,"Hari Sreekumar" <hsreekumar@clickable.com> wrote:

> Hi beneo,
> 
> If you want to just use one machine, why do you want to use hadoop? Hadoop's
> power lies in distributed computing. That being said, it is possible to use
> hadoop on a single machine by using the pseudo-distributed mode (Read
> http://hadoop.apache.org/common/docs/current/single_node_setup.html and
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/).
> If you are using just one machine, at least ensure that your machine has
> lots of cores (8 core/16 cores would be great) to get benefit out of hadoop.
> 
> I am not sure, but using virtual machines won't be helpful here as a virtual
> machine is just an abstraction and not real hardware.
> 
> Cheers,
> Hari
> 
> 2010/11/30 beneo_7 <beneo_7@163.com>
> 
>> i'm sorry, but, are you sure??
>> At 2010-11-30 15:53:58,"rahul patodi" <patodirahul@gmail.com> wrote:
>>> you can create virtual machines on your single machine:
>>> for you have to install sun virtual box(other tools are also available
>> like
>>> VMware)
>>> now you can create as many virtual machine as you want
>>> then you can create one master and all slaves
>>> 
>>> -Thanks and Regards,
>>> Rahul Patodi
>>> Associate Software Engineer,
>>> Impetus Infotech (India) Private Limited,
>>> www.impetus.com
>>> Mob:09907074413
>>> 
>>> 2010/11/30 beneo_7 <beneo_7@163.com>
>>> 
>>>> i have only one machine and it's powerful.
>>>> so, i want the all the slaves and master on one machine?
>>>> 
>>>> thx in advanced
>>>> 
>>> 
>>> 
>>> 
>>> --
>> 
>> 

Mime
View raw message