Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B40D6200CEB for ; Sat, 29 Jul 2017 04:31:41 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B27CE16C092; Sat, 29 Jul 2017 02:31:41 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0414316AE36 for ; Sat, 29 Jul 2017 04:31:40 +0200 (CEST) Received: (qmail 44399 invoked by uid 500); 29 Jul 2017 02:31:40 -0000 Mailing-List: contact commits-help@gobblin.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@gobblin.incubator.apache.org Delivered-To: mailing list commits@gobblin.incubator.apache.org Received: (qmail 44390 invoked by uid 99); 29 Jul 2017 02:31:40 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Jul 2017 02:31:40 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 02A05E361C; Sat, 29 Jul 2017 02:31:40 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: abti@apache.org To: commits@gobblin.apache.org Message-Id: <34265c4e954849d09d51f560e1f90b21@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: incubator-gobblin git commit: [GOBBLIN-174] fix distcp-ng, so it does not remove existing target files Date: Sat, 29 Jul 2017 02:31:40 +0000 (UTC) archived-at: Sat, 29 Jul 2017 02:31:41 -0000 Repository: incubator-gobblin Updated Branches: refs/heads/master 467fe8fc8 -> cec91e610 [GOBBLIN-174] fix distcp-ng, so it does not remove existing target files fixing fullPathdiff() so it does not remove existing target files incase NO_DELETE delete method is specified clearing existingTargetTable if 'existing entity policy' is set to 'update_table' Closes #2008 from arjun4084346/distcpNgFixNoDelete Project: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/commit/cec91e61 Tree: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/tree/cec91e61 Diff: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/diff/cec91e61 Branch: refs/heads/master Commit: cec91e61015568c6afb86c60f27181b7cc346a77 Parents: 467fe8f Author: Arjun Authored: Fri Jul 28 19:31:22 2017 -0700 Committer: Abhishek Tiwari Committed: Fri Jul 28 19:31:22 2017 -0700 ---------------------------------------------------------------------- .../gobblin/data/management/copy/hive/HiveCopyEntityHelper.java | 2 +- .../data/management/copy/hive/UnpartitionedTableFileSet.java | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/cec91e61/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java ---------------------------------------------------------------------- diff --git a/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java b/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java index 230a02a..7cbd7bd 100644 --- a/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java +++ b/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java @@ -619,7 +619,7 @@ public class HiveCopyEntityHelper { if (shouldCopy) { builder.copyFile(sourcePath); } else { - // if not copying, we want to keep the file in the target + // If not copying, we want to keep the file in the target // at the end of this loop, all files in targetExistingPaths will be marked for deletion, so remove this file targetExistingPaths.remove(newPath); desiredTargetExistingPaths.remove(newPath); http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/cec91e61/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/UnpartitionedTableFileSet.java ---------------------------------------------------------------------- diff --git a/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/UnpartitionedTableFileSet.java b/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/UnpartitionedTableFileSet.java index 9d7291a..1481535 100644 --- a/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/UnpartitionedTableFileSet.java +++ b/gobblin-data-management/src/main/java/gobblin/data/management/copy/hive/UnpartitionedTableFileSet.java @@ -70,6 +70,7 @@ public class UnpartitionedTableFileSet extends HiveFileSet { // Update the location of files while keep the existing table entity. log.warn("Source table will not be deregistered while file locaiton has been changed, update source table's" + " file location to" + this.helper.getTargetTable().getDataLocation()); + existingTargetTable = Optional.absent(); break ; case REPLACE_TABLE: // Required to de-register the original table.