hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin <klz...@gmail.com>
Subject Re: DFS. How to read from a specific datanode
Date Thu, 07 Aug 2008 15:11:41 GMT
Yes, I agree with you that it should be negotiated. That is "namenode
provides an ordered list and the client can choose some based on its
own measurements." But I am afraid 0.17.1 does not provide easy
interface for this.


-Kevin



On Thu, Aug 7, 2008 at 3:40 AM, Steve Loughran <stevel@apache.org> wrote:
> Kevin wrote:
>>
>> Thank you for the suggestion. I looked at DFSClient. It appears that
>> chooseDataNode method decides which data node to connect to. Currently
>> it chooses the first non-dead data node returned by namenode, which
>> have sorted the nodes by proximity to the client. However,
>> chooseDataNode is private, so overriding it seems infeasible. Neither
>> are the callers of chooseDataNode public or protected.
>>
>> I need this because I do not want to trust namenode's ordering. For
>> applications where network congestion is rare, we should let the
>> client to decide which data node to load from.
>>
>
> dangerous. what happens when network congestion arrives and the apps are out
> there. Maybe it should be negotiated -namenode provides an ordered list and
> the client can choose some based on its own measurements. If the name node
> provides one only, that's the one you get to use
>

Mime
View raw message