hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11397) When merging expired stripes, we need to create an empty file to preserve metadata.
Date Wed, 25 Jun 2014 09:44:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043255#comment-14043255
] 

Hudson commented on HBASE-11397:
--------------------------------

FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #337 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/337/])
HBASE-11397 When merging expired stripes, we need to create an empty file to preserve metadata
(Victor Xu) (tedyu: rev a280f814144b76d6a909b8f161ca0baa85a151c5)
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeMultiFileWriter.java


> When merging expired stripes, we need to create an empty file to preserve metadata.
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-11397
>                 URL: https://issues.apache.org/jira/browse/HBASE-11397
>             Project: HBase
>          Issue Type: Bug
>          Components: Compaction
>    Affects Versions: 0.98.2
>         Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2
>            Reporter: Victor Xu
>            Assignee: Victor Xu
>             Fix For: 0.99.0, 0.98.4
>
>         Attachments: 11397-v3.patch, HBASE-11397-AssertionError.png, HBASE-11397-HDFS.png,
HBASE-11397-RS-Log.png, HBASE-11397-Stripe-Info.png, HBASE-11397-test.patch, HBASE-11397-v2.patch,
HBASE-11397.patch
>
>
> Stripe Compaction is a good feature in 0.96 and 0.98. But when I used it in a heavy-write
non-uniform row keys scenario(e.g. time dimension in a key), I came across some problems.

> I made my stripes split at the size of 2G(hbase.store.stripe.sizeToSplit=2G), and soon
there were tens of them. It was true that only the last stripe receiving the new keys kept
compacting - old data didn't compact as much, or at all. However, the old stripes were still
there when they all expired. I checked the source code and found that when compacting expired
stripes, the StoreScanner may return no KVs so that SizeMultiWriter.append() is never called.
That's to say, NO NEW FILE WILL BE CREATED. 
> My solution is to create an empty file to preserve metadata at the end of the SizeMultiWriter.commitWritersInternal().



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message