hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14698) Make copyFromLocal's -t option available for put as well
Date Mon, 31 Jul 2017 16:35:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107552#comment-16107552
] 

Mukul Kumar Singh commented on HADOOP-14698:
--------------------------------------------

Thanks for the patch [~boky01], the patch looks good however please take a look at the MoveFromLocal.

Currently multi threading only works for files (i.e. leaf nodes of the tree) while directory
traversing is still single threaded.
If this code is changed, then delete of directory will be tried right after all the files
in the directory are enqueued as tasks.

This will cause the rmdir executed as part of post process to fail because files are still
present in the directory.

{code}
    @Override
    protected void postProcessPath(PathData src) throws IOException {
      if (!src.fs.delete(src.path, false)) {
        // we have no way to know the actual error...
        PathIOException e = new PathIOException(src.toString());
        e.setOperation("remove");
        throw e;
      }
    }
{code}



> Make copyFromLocal's -t option available for put as well
> --------------------------------------------------------
>
>                 Key: HADOOP-14698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14698
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Andras Bokor
>            Assignee: Andras Bokor
>         Attachments: HADOOP-14698.01.patch, HADOOP-14698.02.patch
>
>
> After HDFS-11786 copyFromLocal and put are no longer identical.
> I do not see any reason why not to add the new feature to put as well.
> Being non-identical makes the understanding/usage of command more complicated from user
point of view.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message