hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: data locality on HDFS
Date Sun, 09 May 2010 00:21:34 GMT
Hey Momina,

Here's the path on 20:

DistributedFileSystem#getFileBlockLocations
-> DFSClient#getFileBlockLocations
    -> callGetBlockLocations
       -> ClientProtocol#getBlockLocations
           -> (via proxy) NameNode#getBlockLocations

See createNamenode and createRPCNamenode in the DFSClient constructor
for how the RPC proxy is established.

Thanks,
Eli

On Fri, May 7, 2010 at 8:16 PM, momina khan <momina.azam@gmail.com> wrote:
> hi
>
> i am still going in circles .... i still cant pin point a single
> function call that interacts with the HDFS for block locations... it
> is as if files are making circular calls to getBlockLocations() which
> is implemented such that it calls the same function in a different
> class ... i mean it is not talking to the HDFS anywhere.
>
> plz help!
> momina
>
> On 5/7/10, Amogh Vasekar <amogh@yahoo-inc.com> wrote:
>> Hi,
>> The (o.a.h.fs) FileSystem API has GetBlockLocations that is used to
>> determine replicas.
>> In general cases, (o.a.h.mapreduce.lib.input) FileInputFormat's getSplits()
>> calls this method, which is passed on for job scheduling along with the
>> split info.
>>
>> Hope this is what you were looking for.
>>
>> Amogh
>>
>>
>> On 5/7/10 4:22 PM, "momina khan" <momina.azam@gmail.com> wrote:
>>
>> hi,
>>
>> i am trying to figure out how hadoop uses data locality to schedule maps on
>> nodes which locally store tha map input ... going through code i am going in
>> circles in between a couple of file but not really getting anywhere ... that
>> is to say that i cant locate the HDFS API or func that can communicate a
>> node list that store replicas foe say a block!
>>
>> i am going from FSNameSystem.java to DFSClient.java to
>> BlocksWithLocations.java to DataNodeDescriptor.java and then back again
>> without getting to the HDFS interface that communicates replicas' storing
>> nodes for a block!
>>
>> someone plz help!
>> momina
>>
>>
>

Mime
View raw message