hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akmal Abbasov <akmal.abba...@icloud.com>
Subject Re: HBase memstore flush configurations
Date Fri, 08 May 2015 23:40:22 GMT
Hi Lars,
Thank you for reply.

> We also flush the memstore when the WAL is rolled, which also happens once/hour.

You we’re absolutely right. I had a lot of log rolls in logs.

> Note hbase.regionserver.optionalcacheflushinterval mostly exists to limit the time edits
written with SKIP_WAL should live in the memstore.

The problem in my cluster is that I have a lot of very small flushes.
By small I mean that most of the time they are in Kbytes.
I could live with that, but this causes more serious problem with HBase snapshotting, which
I do each hour.
Each hour memstore flush happens, which could cause a compaction.
After the compaction completes, old hfiles are not deleted, but moved to archive directory,

since I have snapshots referencing to them.
After running for 6 days with default configs, I have my archive directory bigger than data
directory.
So my plan was to flush only in case I have memory pressure(hbase.regionserver.global.memstore.lowerLimit),
and trigger periodical flushes each 24 hours.
Is this approach which will work, or I am doing things wrong.
Is there any best practices for my case?
Thank you.

Kind regards,
Akmal Abbasov

> On 08 May 2015, at 17:16, lars hofhansl <larsh@apache.org> wrote:
> 
> Do you have some log messages from the flushes?
> Maybe you're flushing due to hbase.regionserver.flush.per.changes (number of edit before
we flush).
> 
> We also flush the memstore when the WAL is rolled, which also happens once/hour.
> 
> Note hbase.regionserver.optionalcacheflushinterval mostly exists to limit the time edits
written with SKIP_WAL should live in the memstore.
> 
> 
> We need to make more sense of all these settings!
> 
> 
> -- Lars
> 
> 
> 
> ----- Original Message -----
> From: Akmal Abbasov <akmal.abbasov@icloud.com>
> To: user@hbase.apache.org
> Cc: 
> Sent: Thursday, May 7, 2015 2:40 PM
> Subject: HBase memstore flush configurations
> 
> Hi all, 
> I’ve added the following configurations to my hbase-site.xml file.
> <property>
>    <name>hbase.regionserver.optionalcacheflushinterval</name>
>    <value>86400000</value>
>  </property>
> 
>  <property>
>    <name>hbase.regionserver.global.memstore.upperLimit</name>
>    <value>0.4</value>
>  </property>
> 
>  <property>
>    <name>hbase.regionserver.global.memstore.lowerLimit</name>
>    <value>0.3</value>
>  </property>
> 
> But still I have hourly memstore flushes, which are as I understood the default value
for hbase.regionserver.optionalcacheflushinterval.
> I restarted hbase daemons on all the nodes, including master and regionservers.
> What could be a reason for this?
> Thank you.
> 
> Regards,
> Akmal Abbasov


Mime
View raw message