hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
Date Wed, 04 Oct 2017 11:30:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191144#comment-16191144
] 

Eshcar Hillel commented on HBASE-16417:
---------------------------------------

After a long period of time we experimented with different parameters and in-memory compaction
policies we would like to share with you our main results. 

First set of experiments aims at tuning the Basic (default as of 2.0) in memory compaction
policy. We found out that under zipf distribution its optimal write performance is with active
segment size being 2% of the memory components and with pipeline size bounded to 5 on SSD
and 4 on HDD. The experiments are summarized in the attached pdf file.

Second set of experiments compares a new adaptive compaction policy with the existing basic
and none policies. Bottom line is that there is a tradeoff between write performance and write
volume. We also experimented with mixed read-write workload which show that with in-memory
compaction we are able to reduce the high percentiles of read latency on HDD machines. On
SSD machines we see a minor degradation in read latency. The experiments and main principle
of the adaptive policy are also summarized in an attached pdf file.

I am attaching a patch for trunk which includes the new adaptive policy as well as the new
optimal parameters for basic.

I would be happy to give more details on the experiment settings and results and answer any
questions you may have.
And also waiting to get your comments on reviewboard :)
Thank you!

> In-Memory MemStore Policy for Flattening and Compactions
> --------------------------------------------------------
>
>                 Key: HBASE-16417
>                 URL: https://issues.apache.org/jira/browse/HBASE-16417
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Anastasia Braginsky
>            Assignee: Eshcar Hillel
>             Fix For: 3.0.0
>
>         Attachments: HBASE-16417 - Adaptive Compaction Policy - 20171001.pdf, HBASE-16417-benchmarkresults-20161101.pdf,
HBASE-16417-benchmarkresults-20161110.pdf, HBASE-16417-benchmarkresults-20161123.pdf, HBASE-16417-benchmarkresults-20161205.pdf,
HBASE-16417-benchmarkresults-20170309.pdf, HBASE-16417-benchmarkresults-20170317.pdf, HBASE-16417
- parameter tuning - 20171001.pdf, HBASE-16417-V01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message