hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: multiple datanodes in the same machine
Date Tue, 15 Apr 2008 22:24:26 GMT

Then the answer is yes.  You need 10 configuration directories.


On 4/15/08 3:12 PM, "Cagdas Gerede" <cagdas.gerede@gmail.com> wrote:

> I am working on Distributed File System part. I do not use MR part,
> and I need to run multiple processes to test some scenarios on the file
> system.
> 
> On Tue, Apr 15, 2008 at 1:37 PM, Ted Dunning <tdunning@veoh.com> wrote:
> 
>> 
>> I have had no issues in scaling the number of datanodes.  The location of
>> the data is almost invisible to MR programs.
>> 
>> I have had issues in going from local to distributed mode, but that has
>> entirely been due to class path like issues.  Since MR naturally restricts
>> your focus, it is pretty much the rule that programs scale without much
>> thought.
>> 
>> If you test with two tasktrackers and one data node, you should have a
>> pretty solid test environment.
>> 
>> 
>> On 4/15/08 1:12 PM, "cagdas.gerede@gmail.com" <cagdas.gerede@gmail.com>
>>  wrote:
>> 
>>> Testing when I do not have 10 machines.
>>> 
>>> 
>>> On 4/15/08, Ted Dunning <tdunning@veoh.com> wrote:
>>>> 
>>>> Why do you want to do this perverse thing?
>>>> 
>>>> How does it help to have more than one datanode per machine?  And what
>> in
>>>> the world is better when you have 10?
>>>> 
>>>> 
>>>> On 4/15/08 12:53 PM, "Cagdas Gerede" <cagdas.gerede@gmail.com> wrote:
>>>> 
>>>>> I have a follow-up question,
>>>>> Is there a way to programatically configure datanode parameters and
>> start
>>>>> the datanode process?
>>>>> If I want to create 10 datanodes on the same host, do I have to create
>> 10
>>>>> config files?
>>>>> 
>>>>> 
>>>>> On Tue, Apr 15, 2008 at 12:29 PM, dhruba Borthakur <
>> dhruba@yahoo-inc.com>
>>>>> wrote:
>>>>> 
>>>>>> Yes, just point the Datanodes to different config files, different
>> sets
>>>>>> of ports, different data directories. Etc.etc.
>>>>>> 
>>>>>> Thanks,
>>>>>> dhruba
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Cagdas Gerede [mailto:cagdas.gerede@gmail.com]
>>>>>> Sent: Tuesday, April 15, 2008 11:21 AM
>>>>>> To: core-user@hadoop.apache.org
>>>>>> Subject: multiple datanodes in the same machine
>>>>>> 
>>>>>> Is there a way to run multiple datanodes in the same machine?
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> ------------
>>>>>> Best Regards, Cagdas Evren Gerede
>>>>>> Home Page: http://cagdasgerede.info
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
> 


Mime
View raw message