Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.223.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <E7A0550E15952B4398C403EE236A6DEF87F4D6@chnmbx02.apac.sas.com>
References: <E7A0550E15952B4398C403EE236A6DEF87F4D6@chnmbx02.apac.sas.com>
From: Harsh J <harsh@cloudera.com>
Date: Thu, 6 Sep 2012 18:35:59 +0530
Message-ID: 
 <CAOcnVr3eabBum10tmUKTZHgU4GG8W7PYJ3PPxFMex=hMBnQm1g@mail.gmail.com>
Subject: Re: Transfer blocks from one datanode to another
To: hdfs dev <hdfs-dev@hadoop.apache.org>
Cc: adrian.liu@sas.com
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,

Please do not use the general@ lists for development/usage questions.
This list is meant for project-level discussions alone. Thanks! :)

I've moved this mail to hdfs-dev@hadoop.apache.org. When replying,
please instead use this list, going forward.

My reply inline:

On Thu, Sep 6, 2012 at 3:12 PM, Adrian (Xinyu) Liu <adrian.liu@sas.com> wro=
te:
> Hi All,
>
> Nowadays, I am working with HDFS and implementing some functionalities ba=
se on HDFS API.
> As I knew, one specific file is divided into several blocks and distribut=
ed into different datanode with certain replication numbers.
> And I want to find out a series of HDFS API which can meet the following =
requirement:
>
>
> 1.       Given a specific filename and related information that already u=
ploaded into the HDFS, retrieve: how many blocks are there,
>
> each datanode contain which blocks, etc.

This isn't possible to get if you're using simple Public APIs.

The FileSystem#getFileBlockLocations will tell you what hosts are
carrying the blocks of a file (a list of hosts for each block in the
file). See http://hadoop.apache.org/common/docs/current/api/org/apache/hado=
op/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,%20lo=
ng,%20long)

For the list of block IDs, you'd have to pull from a DFSClient
instance, which calls the (NameNode-side) ClientProtocol's
getBlockLocations(=85) method call. See the interface at
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-pr=
oject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProto=
col.java?view=3Dmarkup

> 2.       Given a specific filename, source datanode, specific block id, d=
estination datanode and related information to transfer the block
>
> from source node to destination node.

This needs to be done via the DataTransferProtocol, and its specific
method of replaceBlock(=85). See the interface at
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-pr=
oject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfe=
r/DataTransferProtocol.java?view=3Dmarkup

> I've read several materials and API reference about HDFS and can't find a=
ppropriate ones, the objective of this mail is to make sure
> if such  API existed, and if so, what are they (especially the second one=
, transfer a specific block of a specific file from a certain datanode to a=
nother)
>
> There is a tool called Balancer already existed in HDFS package, I am rea=
ding the source code, but it's too intricate to track the line, can anyone =
help me?

In the Balancer sources, see the final replaceBlock(=85) call made at
L376 at http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop=
-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balan=
cer/Balancer.java?view=3Dmarkup,
and then trace backwards from that point to see how its built up till
that point.

Feel free to send across any more questions you have!

--=20
Harsh J