hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3404) Compaction Ordering for Bulk Import Files
Date Fri, 31 Dec 2010 00:09:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976153#action_12976153

Todd Lipcon commented on HBASE-3404:

I thought we had decided at some point that this ordering didn't matter anymore? If so, could
we sort the storefiles by size before deciding which files to compact?

> Compaction Ordering for Bulk Import Files
> -----------------------------------------
>                 Key: HBASE-3404
>                 URL: https://issues.apache.org/jira/browse/HBASE-3404
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.90.0, 0.90.1, 0.92.0
>            Reporter: Nicolas Spiegelberg
>            Assignee: Nicolas Spiegelberg
> We got into an issue today where we were using HFileOutputFormat to perform an incremental
load on an already-large cluster.  Because bulk-loaded files don't have a sequence ID, they
are put in the front of the StoreFile list.  This resulted in the following StoreFile ordering
> 2GB (bulk) => 25GB => 2GB => ...
> So this triggered a 30+GB major compaction for every single region.  Optimally, we would
like bulk import files to be ordered in the compaction list at the time of insertion so this
can be a much smaller compaction and rely on StoreFile age for major compaction trigger.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message