hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Segel, Mike" <mse...@navteq.com>
Subject Re: Stability issue - dead DN's
Date Fri, 13 May 2011 12:36:53 GMT
Bonded will work but you may not see the performance you would expect.  If you need >1 GBe,
go 10GBe less headache and has even more headroom.

Multiple interfaces won't work. Or I should say didn't work in past releases. 
If you think about it, clients have to connect to each node. So having two interfaces and
trying to manage them makes no sense. 

Add to this trying to manage this in DNS ... Why make more work for yourself?
Going from memory... It looked like you rDNS had to match you hostnames so your internal interfaces
had to match hostnames so you had an inverted network.

If you draw out your network topology you end up with a ladder. 
You would be better off (IMHO) to create a subnet where only your edge servers are dual nic'd.
But then if your cluster is for development... Now your PCs can't be used as clients...

Does this make sense?


Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 4:57 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wrote:

> Hi Mike,
> 
>> You really really don't want to do this.
>> Long story short... It won't work.
> 
> Can you elaborate? Are you talking about the bonded interfaces or about having a separated
network for interconnects and external network? What can go wrong there?
> 
>> 
>> Just a suggestion.. You don't want anyone on your cluster itself. They
>> should interact wit edge nodes, which are 'Hadoop aware'. Then your
>> cluster has a single network to worry about.
> 
> That's our current setup. We have a single headnode that is used as a SPOE. However,
I'd like to change that on our future production system. We want to implement Kerberos for
authentication, and let users interact with the cluster from their own machine. This would
enable them to submit their jobs from the local IDE. The only way to do this is by opening
up Hadoop ports for the world, is my understanding: if people interact with HDFS they need
to be able to interact with all nodes, right? What would be the argument against this?
> 
> Cheers,
> Evert
> 
>> 
>> 
>> Sent from a remote device. Please excuse any typos...
>> 
>> Mike Segel
>> 
>> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>> 
>>> 
>>> 
>>> 
>>> 
>>>>> * a 2x1GE bonded network interface for interconnects
>>>>> * a 2x1GE bonded network interface for external access
>>> 
>>>   Multiple NICs on a box can sometimes cause big performance
>> problems with Hadoop.  So watch your traffic carefully.
>>> 
>>> 
>>> 


The information contained in this communication may be CONFIDENTIAL and is intended only for
the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this communication in error, please
notify the sender and delete/destroy the original message and any copy of it from your computer
or paper files.

Mime
View raw message