Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=w7+PkxYonfVKWzemDz1j/jbwz7fEIoFtfsO3kWDms0vREIBzYkTh8aB341T62Wya
Message-ID: <4A9340FB.8030200@yahoo-inc.com>
Date: Mon, 24 Aug 2009 18:40:11 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: hdfs-user@hadoop.apache.org
Subject: Re: Question about HDFS Architecture
References: <30903.1416.qm@web51001.mail.re2.yahoo.com>
In-Reply-To: <30903.1416.qm@web51001.mail.re2.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Harold,

Both answers by Aaron were incorrect.

 > Does the client cache this information, or does it always talk to the namenode first?

Yes, the client caches replica locations received from the name-node.
On open() it receives locations of the first 10 blocks of the file.
In most cases these are all file blocks. If not then the client will
get another portion of blocks when needed, and will also cache them.

 > Also, if a file has multiple replicas stored on multiple datanodes on the same "rack", how does the namenode pick which datanode the client has to talk to?

The name-node returns block locations ordered by the proximity to the client.
The client always contacts data-nodes in this order. It cannot make any decisions
about the proximity because it does not possess knowledge about the cluster topology.
If all replicas are on the same rack but not local to the client then the ordering
returned by the name-node is arbitrary.
This may happen mostly if network topology is not configured.
Otherwise replicas should be distributed on different racks.
3 replicas should be on at least 2 racks.

Thanks
--Konstantin


Harold Lim wrote:
> To read/get a file, I understand that a client first contacts the namenode to determine which datanode has the file/block. Then, it contacts the datanode for the actual file.
> 
> Does the client cache this information, or does it always talk to the namenode first? 
> 
> Also, if a file has multiple replicas stored on multiple datanodes on the same "rack", how does the namenode pick which datanode the client has to talk to? In this case, all datanodes are homogeneous, which makes the "rack-awareness" unimportant to the decision making.
> 
> Thanks,
> Harold
> 
> 
>       
>