hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (HADOOP-15208) DistCp to offer -xtrack <path> option to save src/dest filesets as alternative to delete()
Date Thu, 22 Mar 2018 05:29:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kumar Vavilapalli reopened HADOOP-15208:
----------------------------------------------

Reopening and closing this instead as a dup of HADOOP-15209 as I can't find any patch for
this in 3.1.0 for this JIRA. Revert back if this is incorrect.

> DistCp to offer -xtrack <path> option to save src/dest filesets as alternative
to delete()
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15208
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools/distcp
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>             Fix For: 3.1.0
>
>         Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, HADOOP-15208-002.patch,
HADOOP-15208-003.patch
>
>
> There are opportunities to improve distcp delete performance and scalability with object
stores, but you need to test with production datasets to determine if the optimizations work,
don't run out of memory, etc.
> By adding the option to save the sequence files of source, dest listings, people (myself
included) can experiment with different strategies before trying to commit one which doesn't
scale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message