From dev-return-146303-archive-asf-public=cust-asf.ponee.io@hive.apache.org Tue Feb 13 22:36:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3F84218067B for ; Tue, 13 Feb 2018 22:36:06 +0100 (CET) Received: (qmail 25572 invoked by uid 500); 13 Feb 2018 21:36:05 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 25559 invoked by uid 99); 13 Feb 2018 21:36:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Feb 2018 21:36:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B5C2C180166 for ; Tue, 13 Feb 2018 21:36:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id p7fcBjgGt6dR for ; Tue, 13 Feb 2018 21:36:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 643CC5F39C for ; Tue, 13 Feb 2018 21:36:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5E3F1E02C9 for ; Tue, 13 Feb 2018 21:36:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 54F2124122 for ; Tue, 13 Feb 2018 21:36:00 +0000 (UTC) Date: Tue, 13 Feb 2018 21:36:00 +0000 (UTC) From: "Eugene Koifman (JIRA)" To: dev@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HIVE-18709) Enable Compaction to work on more than one partition per job MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Eugene Koifman created HIVE-18709: ------------------------------------- Summary: Enable Compaction to work on more than one partition per job Key: HIVE-18709 URL: https://issues.apache.org/jira/browse/HIVE-18709 Project: Hive Issue Type: Improvement Components: Transactions Affects Versions: 1.0.0 Reporter: Eugene Koifman Assignee: Eugene Koifman currently compaction launches 1 MR job per partition that needs to be compacted. The number of tasks is equal to the number of buckets in the table (or number or writers in the 'widest' write). The number of AMs in a cluster is usually limited to a small percentage of the nodes. This limits how much compaction can be done in parallel. Investigate what it would take for a single job to be able to handle multiple partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)