Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 85A2B200CB7 for ; Fri, 30 Jun 2017 21:09:03 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 829C4160BF6; Fri, 30 Jun 2017 19:09:03 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C8E58160BE8 for ; Fri, 30 Jun 2017 21:09:02 +0200 (CEST) Received: (qmail 58632 invoked by uid 500); 30 Jun 2017 19:09:02 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 58622 invoked by uid 99); 30 Jun 2017 19:09:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jun 2017 19:09:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A83901AFA6F for ; Fri, 30 Jun 2017 19:09:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 8nQdyA7WYid5 for ; Fri, 30 Jun 2017 19:09:00 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id C41425F46F for ; Fri, 30 Jun 2017 19:09:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 577A0E0026 for ; Fri, 30 Jun 2017 19:09:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0EAF4245CC for ; Fri, 30 Jun 2017 19:09:00 +0000 (UTC) Date: Fri, 30 Jun 2017 19:09:00 +0000 (UTC) From: "Sankar Hariappan (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-16901) Distcp optimization - One distcp per ReplCopyTask MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 30 Jun 2017 19:09:03 -0000 [ https://issues.apache.org/jira/browse/HIVE-16901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16901: ------------------------------------ Status: Open (was: Patch Available) > Distcp optimization - One distcp per ReplCopyTask > -------------------------------------------------- > > Key: HIVE-16901 > URL: https://issues.apache.org/jira/browse/HIVE-16901 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl > Affects Versions: 2.1.0 > Reporter: Sankar Hariappan > Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16901.01.patch > > > Currently, if a ReplCopyTask is created to copy a list of files, then distcp is invoked for each and every file. Instead, need to pass the list of source files to be copied to distcp tool which basically copies the files in parallel and hence gets lot of performance gain. -- This message was sent by Atlassian JIRA (v6.4.14#64029)