hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19528) Major Compaction Tool
Date Tue, 06 Feb 2018 00:01:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353118#comment-16353118

stack commented on HBASE-19528:

Failures are compiling thrift... Have seen them before. Unrelated to hadoop...

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.2:install
(default-install) on project hbase-thrift: Failed to install metadata org.apache.hbase:hbase-thrift:3.0.0-SNAPSHOT/maven-metadata.xml:
Could not parse metadata /home/jenkins/.m2/repository/org/apache/hbase/hbase-thrift/3.0.0-SNAPSHOT/maven-metadata-local.xml:
in epilog non whitespace content is not allowed but got / (position: END_TAG seen ...</metadata>\n/...
@25:2)  -> [Help 1]

But it passed for the versions that failed with this patch.

Let me try again.

> Major Compaction Tool 
> ----------------------
>                 Key: HBASE-19528
>                 URL: https://issues.apache.org/jira/browse/HBASE-19528
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: churro morales
>            Assignee: churro morales
>            Priority: Major
>             Fix For: 2.0.0-beta-2
>         Attachments: 0001-HBASE-19528-Do-nothing-patch.patch, 0001-HBASE-19528-Do-nothing-patch.patch,
0001-HBASE-19528-Major-Compaction-Tool-ADDENDUM.patch, HBASE-19528.branch-1.patch, HBASE-19528.patch,
HBASE-19528.v1.branch-1.patch, HBASE-19528.v1.patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch,
HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v8.patch
> The basic overview of how this tool works is:
> Parameters:
>     Table
>     Stores
>     ClusterConcurrency
>     Timestamp
> So you input a table, desired concurrency and the list of stores you wish to major compact.
 The tool first checks the filesystem to see which stores need compaction based on the timestamp
you provide (default is current time).  It takes that list of stores that require compaction
and executes those requests concurrently with at most N distinct RegionServers compacting
at a given time.  Each thread waits for the compaction to complete before moving to the next
queue.  If a region split, merge or move happens this tool ensures those regions get major
compacted as well. 
> This helps us in two ways, we can limit how much I/O bandwidth we are using for major
compaction cluster wide and we are guaranteed after the tool completes that all requested
compactions complete regardless of moves, merges and splits. 

This message was sent by Atlassian JIRA

View raw message