Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 18 Aug 2017 22:19:01 +0000 (UTC)
From: "Kai Zheng (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13090905.1501277168000.68811.1503094741867@Atlassian.JIRA>
In-Reply-To: <JIRA.13090905.1501277168000@Atlassian.JIRA>
References: <JIRA.13090905.1501277168000@Atlassian.JIRA> <JIRA.13090905.1501277168489@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-12222) Add EC information to BlockLocation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 18 Aug 2017 22:19:11 -0000


    [ https://issues.apache.org/jira/browse/HDFS-12222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133699#comment-16133699 ] 

Kai Zheng commented on HDFS-12222:
----------------------------------

bq. Also, if someone is going below DFSClient to read, they can also get the cellSize from the LocatedBlock/HdfsFileStatus information.
Yes, agree. It has to be that, since much more info would be needed to support that kind of hack and only HDFS specific API can provide.

So looks like what's exactly needed would be just having the existing API return all the data block locations in EC case instead of the replication locations. The location info can be used to support data processing scheduling, pretty enough and clean. Cool!

> Add EC information to BlockLocation
> -----------------------------------
>
>                 Key: HDFS-12222
>                 URL: https://issues.apache.org/jira/browse/HDFS-12222
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
>            Assignee: Huafeng Wang
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-12222.001.patch
>
>
> HDFS applications query block location information to compute splits. One example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
>     long bytesInThisBlock = blkLocations[startIndex].getOffset() + 
>                           blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as well, which are not part of the logical file length. This messes up the offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy and then parsing the schema, but it'd be a lot better if we exposed this more generically in BlockLocation instead.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org