hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3873) DistCp should have an option for limiting the number of files/bytes being copied
Date Fri, 08 Aug 2008 19:57:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621032#action_12621032

Raghu Angadi commented on HADOOP-3873:

This is a useful feature. Hopefully documentation clearly defines what users can expect.

One question : extending the example in the 2nd commend above, what happens if  /user1/data/file1
is deleted on the source before the second day? will it be deleted on the destination? If
yes, may be some option like "-sync" will make it more clear to the user (of course "-sizelimit"
etc still apply). 

In the long term, once we can preserve modification times and other metadata while copying,
it might be better to add "rsync" mode to distcp.

> DistCp should have an option for limiting the number of files/bytes being copied
> --------------------------------------------------------------------------------
>                 Key: HADOOP-3873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3873
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: tools/distcp
>            Reporter: Tsz Wo (Nicholas), SZE
> A single DistCp command may potentially copies a huge number of files/bytes.  In such
case, DistCp will run a long time and there is no way stop it nicely.  It would be good if
DistCp have an option to limit the number of files/bytes being copied.  Once the limit is
reached, DistCp will terminate and return success.  All files copied are guaranteed to be
good and there is no partially copied file.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message