hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
Date Wed, 07 Feb 2018 21:26:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356104#comment-16356104
] 

Steve Loughran commented on HADOOP-15208:
-----------------------------------------

Patch 002
 * address checkstyle issues
 * remove that new HDFS test because it has fulfilled its purpose (made sure the abstract
test suite was getting paths correct when local fs != remote fs without having to use s3 or
azure suites). Most of the other distcp tests use HDFS as their test cluster anyway, and the
new tests were slow (cluster setup/teardown, time to upload files)
* Testing: S3A US-West, Azure Ireland

Still undocumented. I hope to leave it that way

> DistCp to offer option to save src/dest filesets as alternative to delete()
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-15208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15208
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools/distcp
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch
>
>
> There are opportunities to improve distcp delete performance and scalability with object
stores, but you need to test with production datasets to determine if the optimizations work,
don't run out of memory, etc.
> By adding the option to save the sequence files of source, dest listings, people (myself
included) can experiment with different strategies before trying to commit one which doesn't
scale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message