From issues-return-332616-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Tue Feb 6 06:36:12 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id BE0D6180657 for ; Tue, 6 Feb 2018 06:36:12 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id A118C160C5B; Tue, 6 Feb 2018 05:36:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F1CF4160C3B for ; Tue, 6 Feb 2018 06:36:11 +0100 (CET) Received: (qmail 48601 invoked by uid 500); 6 Feb 2018 05:36:11 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 48579 invoked by uid 99); 6 Feb 2018 05:36:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Feb 2018 05:36:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 33C221A06C4 for ; Tue, 6 Feb 2018 05:36:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -102.311 X-Spam-Level: X-Spam-Status: No, score=-102.311 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 4heH6E_lnM_m for ; Tue, 6 Feb 2018 05:36:09 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id A53505F2AB for ; Tue, 6 Feb 2018 05:36:08 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7E21CE00EA for ; Tue, 6 Feb 2018 05:36:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id E007121E82 for ; Tue, 6 Feb 2018 05:36:01 +0000 (UTC) Date: Tue, 6 Feb 2018 05:36:00 +0000 (UTC) From: "stack (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-19528) Major Compaction Tool MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353369#comment-16353369 ] stack commented on HBASE-19528: ------------------------------- I call the above a green build. The test failure is unrelated and the three hadoop fails are different to the original set and looking in logs are the thrift fetch issue, not hadoop compile... One more run while we're sleeping... > Major Compaction Tool > ---------------------- > > Key: HBASE-19528 > URL: https://issues.apache.org/jira/browse/HBASE-19528 > Project: HBase > Issue Type: New Feature > Reporter: churro morales > Assignee: churro morales > Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19528-Do-nothing-patch.patch, 0001-HBASE-19528-Do-nothing-patch.patch, 0001-HBASE-19528-Major-Compaction-Tool-ADDENDUM.patch, HBASE-19528.branch-1.patch, HBASE-19528.patch, HBASE-19528.v1.branch-1.patch, HBASE-19528.v1.patch, HBASE-19528.v2.branch-1 (1).patch, HBASE-19528.v2.branch-1 (1).patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v8.patch > > > The basic overview of how this tool works is: > Parameters: > Table > Stores > ClusterConcurrency > Timestamp > So you input a table, desired concurrency and the list of stores you wish to major compact. The tool first checks the filesystem to see which stores need compaction based on the timestamp you provide (default is current time). It takes that list of stores that require compaction and executes those requests concurrently with at most N distinct RegionServers compacting at a given time. Each thread waits for the compaction to complete before moving to the next queue. If a region split, merge or move happens this tool ensures those regions get major compacted as well. > This helps us in two ways, we can limit how much I/O bandwidth we are using for major compaction cluster wide and we are guaranteed after the tool completes that all requested compactions complete regardless of moves, merges and splits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)