hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded
Date Tue, 11 Oct 2016 18:04:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15566111#comment-15566111
] 

Zhe Zhang commented on HDFS-10971:
----------------------------------

Thanks for the discussions Andrew / Wei-Chiu.

For the initial 3.0 release I prefer a simpler strategy of only supporting {{-pr}} when both
the source and destination directories are replicated. 

{quote}
-setPolicy \[-p <policyName>\] <path>

Sets an ErasureCoding policy on a directory at the specified path.
path: An directory in HDFS. This is a mandatory parameter. Setting a policy only affects newly
created files, and does not affect existing files.
{quote}

The current EC policy protocol is that if a directory is configured with a policy, it will
be applied to all newly created direct children files under it. Extending this protocol will
complicate how users/admins understand EC policies, and probably needs more discussions (maybe
after the EC feature has been used in production for a while).

Thoughts?

> Distcp should not copy replication factor if source file is erasure coded
> -------------------------------------------------------------------------
>
>                 Key: HDFS-10971
>                 URL: https://issues.apache.org/jira/browse/HDFS-10971
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: distcp
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to store erasure
coding policy.
> Distcp copies the source file's replication factor to the destination if {{-pr}} is specified.
However, if the source file is EC, the replication factor (which is EC policy) should not
be replicated to the destination file. When a HdfsFileStatus is converted to FileStatus, the
replication factor is set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the replication factor
of an EC file results in an IOException: "Requested replication factor of 0 is less than the
required minimum of 1 for /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message