hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Can a jobtracker directly access datanode information (block placement) on namenode?
Date Sat, 21 Jul 2012 03:54:31 GMT

Any HDFS client can request a list of block locations for a given file
path (node-level detail of where blocks are placed for a file), via
the FileSystem#getFileBlockLocations API:

MR too gets this info via the user's InputFormat#getSplits method, and
schedules with these locations.

On Sat, Jul 21, 2012 at 2:06 AM, Kyungyong Lee <iamkyungyong@gmail.com> wrote:
> Hello all,
> I want to get datanode information (related to block placement) that is kept
> at a namenode from a jobtracker. As far as I understand, the jobtracker uses
> the locality-of-data for job scheduling, so I believe the jobtracker is
> keeping the information somewhere in the source code. However, I could not
> find the location. Can anyone give me a starting point (source code) where
> the jobtracker has access to block placement information? Thanks.

Harsh J

View raw message