hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das" <d...@yahoo-inc.com>
Subject RE: Hadoop topology.script.file.name Form
Date Mon, 09 Jun 2008 08:20:12 GMT
This documentation is for the earlier versions. In 0.17 the way in which
racks are dealt with has changed.

> -----Original Message-----
> From: Yang Chen [mailto:chenyangyinpeng@gmail.com] 
> Sent: Sunday, June 08, 2008 8:06 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Hadoop topology.script.file.name Form
> 
> Rack Awareness
> 
> Typically large Hadoop clusters are arranged in *racks* and 
> network traffic between different nodes with in the same rack 
> is much more desirable than network traffic across the racks. 
> In addition Namenode tries to place replicas of block on 
> multiple racks for improved fault tolerance. Hadoop lets the 
> cluster administrators decide which *rack* a node belongs to 
> through configuration variable dfs.network.script. When this 
> script is configured, each node runs the script to determine 
> its *rackid*. A default installation assumes all the nodes 
> belong to the same rack. This feature and configuration is 
> further described in 
> PDF<http://issues.apache.org/jira/secure/attachment/12345251/R
> ack_aware_HDFS_proposal.pdf>attached
> to
> HADOOP-692 <http://issues.apache.org/jira/browse/HADOOP-692>.
> 
> Hope this will be helpful.
> 
> 
> 
> YC
> 
> On Sun, Jun 8, 2008 at 9:53 PM, Devaraj Das 
> <ddas@yahoo-inc.com> wrote:
> 
> > Hi Iver,
> > The implementation of the script depends on your setup. The 
> main thing 
> > is that it should be able to accept a bunch of IP addresses and DNS 
> > names and be able to give back the rackIDs for each. It is a 
> > one-to-one correspondence between what you pass and what 
> you get back. 
> > For getting the rackID the script could deduce it from the 
> IP address, 
> > or, query some service (similar to the way dns works, or 
> some similar 
> > mechanism), or, in the extreme case, read a file that has 
> the mapping 
> > from IP address to rackId.
> > Thanks,
> > Devaraj.
> >
> > > -----Original Message-----
> > > From: ?? [mailto:tianchao@software.ict.ac.cn]
> > > Sent: Friday, June 06, 2008 8:13 AM
> > > To: core-user
> > > Subject: Hadoop topology.script.file.name Form
> > >
> > > hi,
> > >
> > > I want to setup a hadoop cluster, and I want to make the 
> cluster to 
> > > be Rack Awareness. But I can't find any document about 
> the form of 
> > > topology.script.file.name.
> > >
> > > Could anybody give me an example about the form of 
> > > topology.script.file.name?
> > >
> > > thanks a lot.
> > >
> > > iver
> > >
> > > 2008-06-06
> > > ________________________________
> > >
> > >
> > >
> > >
> >
> >
> 


Mime
View raw message