hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Austin Heyne <ahe...@ccri.com>
Subject Re: Stripe Compactions Stability
Date Wed, 19 Sep 2018 21:39:32 GMT
Thanks for the info. Looks like I'll avoid using that feature then. We 
have a fairly complicated and well thought out compaction management 
running but since we're moving into needing near 100% uptime we don't 
have the window for tables to be tied up compacting. We're working on a 
solution to chunk our data up over multiple tables so it's more 
manageable. However, we'll be stuck with the large table until it done 
aging off a year from now.

I'll look into the tiered compactions but we have a spatio-temporal key 
(GeoMesa) and we may need to write an extension to the 
DateTieredStoreEngine.

Thanks,
Austin


On 09/17/2018 07:51 PM, sudhir patil wrote:
> Have you considered below options to reduce number/time of compactions?
>
> - Reduce memstore flushes to create fewer hfiles there by reducing number
> of compactions. By setting hbase.hregion.memstore.flush.size and other
> parameters.
> - hbase.offpeak.start & end , so that compactions triggered during off peak
> time
> - disabling major compaction and running manually on days with low load OR
> delay major compactions using *hbase.hregion.majorcompaction*
> - Checkout date tiered compaction, which doesn't compact older hfiles
> https://issues.apache.org/jira/browse/HBASE-15337
>
> On Tue, Sep 18, 2018 at 6:53 AM Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Hi,
>> To my knowledge, stripe compaction has not seen patches for a few years.
>>
>> Have you looked at :
>> http://hbase.apache.org/book.html#ops.date.tiered
>>
>> If the above doesn't suit your needs, can you tell us more about your use
>> case ?
>>
>> Thanks
>>
>> On Mon, Sep 17, 2018 at 11:39 AM Austin Heyne <aheyne@ccri.com> wrote:
>>
>>> The HBase cluster we're running has well over 100TB of data spread
>>> across two tables and as you'd guess we're suffering from compaction
>>> times taking way to long (and we need to double or triple that volume).
>>> I've found information on Stripe Compactions [1] which it seems like we
>>> could benefit a lot from, however, 'experimental' is a scary word for
>>> productions systems so I just wanted to get a general sentiment around
>>> the stability of the feature. Any experience or input would be very
>>> helpful.
>>>
>>> Thanks again,
>>> Austin
>>>
>>> [1] https://hbase.apache.org/book.html#ops.stripe
>>>
>>>

-- 
Austin L. Heyne


Mime
View raw message