hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded
Date Thu, 06 Oct 2016 22:06:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553330#comment-15553330
] 

Andrew Wang commented on HDFS-10971:
------------------------------------

1, based on this code from S3AFileStatus:

{code}
  // Files
  public S3AFileStatus(long length, long modification_time, Path path,
      long blockSize) {
    super(length, false, 1, blockSize, modification_time, path);
    isEmptyDirectory = false;
  }
{code}

I think this is okay as long as we properly expose the durability level of EC files elsewhere.
I've heard of operators for instance who want to verify there aren't any under-replicated
files when doing maintenance operations.

> Distcp should not copy replication factor if source file is erasure coded
> -------------------------------------------------------------------------
>
>                 Key: HDFS-10971
>                 URL: https://issues.apache.org/jira/browse/HDFS-10971
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: distcp
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to store erasure
coding policy.
> Distcp copies the source file's replication factor to the destination if {{-pr}} is specified.
However, if the source file is EC, the replication factor (which is EC policy) should not
be replicated to the destination file. When a HdfsFileStatus is converted to FileStatus, the
replication factor is set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the replication factor
of an EC file results in an IOException: "Requested replication factor of 0 is less than the
required minimum of 1 for /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message