Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8E15D200D5C for ; Fri, 15 Dec 2017 23:24:11 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 8C34D160C26; Fri, 15 Dec 2017 22:24:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D25E9160C14 for ; Fri, 15 Dec 2017 23:24:10 +0100 (CET) Received: (qmail 45357 invoked by uid 500); 15 Dec 2017 22:24:09 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 45295 invoked by uid 99); 15 Dec 2017 22:24:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Dec 2017 22:24:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 8B717C1BDD for ; Fri, 15 Dec 2017 22:24:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.011 X-Spam-Level: X-Spam-Status: No, score=-100.011 tagged_above=-999 required=6.31 tests=[SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id thPQQbMhImJh for ; Fri, 15 Dec 2017 22:24:08 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 91E3B5FB54 for ; Fri, 15 Dec 2017 22:24:07 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 40D1DE256E for ; Fri, 15 Dec 2017 22:24:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 98FC82740C for ; Fri, 15 Dec 2017 22:24:01 +0000 (UTC) Date: Fri, 15 Dec 2017 22:24:01 +0000 (UTC) From: "churro morales (JIRA)" To: dev@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HBASE-19528) Major Compaction Tool MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 15 Dec 2017 22:24:11 -0000 churro morales created HBASE-19528: -------------------------------------- Summary: Major Compaction Tool Key: HBASE-19528 URL: https://issues.apache.org/jira/browse/HBASE-19528 Project: HBase Issue Type: New Feature Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 3.0.0 The basic overview of how this tool works is: Parameters: Table Stores ClusterConcurrency Timestamp So you input a table, desired concurrency and the list of stores you wish to major compact. The tool first checks the filesystem to see which stores need compaction based on the timestamp you provide (default is current time). It takes that list of stores that require compaction and executes those requests concurrently with at most N distinct RegionServers compacting at a given time. Each thread waits for the compaction to complete before moving to the next queue. If a region split, merge or move happens this tool ensures those regions get major compacted as well. This helps us in two ways, we can limit how much I/O bandwidth we are using for major compaction cluster wide and we are guaranteed after the tool completes that all requested compactions complete regardless of moves, merges and splits. -- This message was sent by Atlassian JIRA (v6.4.14#64029)