hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
Date Thu, 09 Mar 2017 15:11:38 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903176#comment-15903176

Eshcar Hillel commented on HBASE-16417:


We focus on small data (100B values), uniform and zipfian distribution of keys, Async and
sync wal modes.
We see excellent results for uniform distribution and good results for zipfian distribution
in write only mode, and fair improvement in read latencies in mixed workload.

Summary of results:
Write only
We run write-only benchmarks with zipfian and uniform distribution of keys.
* Uniform distribution: basic outperforms no compaction by 40%; eager improves throughput
of no compaction by 10%, however the 99th latency percentile in eager is slower by 18% than
no compaction. This is likely due to unneeded in-memory compactions occurring at the background.

Basic and eager improve write amplification by 25% and 15% respectively. 
* Zipfian distribution: when running in async wal mode eager improves write amplification
by 30%. Basic and eager outperform no compaction by 20% and 15%, resp. This can be attributed
in part to running less GC and in part to doing less IO. When running sync wal
mode eager improves write amplification by 10%. Other than that all policies are comparable.
Basic and eager slightly improve over no compaction. 
In sync wal mode the throughput is much lower. Async wal mode represents a scenario where
the system is loaded by many clients with much higher load.

Mixed workload
Eager improves over no compaction by 6-10%. Basic improves the 50th percentile by 7% but the
performance of 95th and 99th percentile degrade the performance by 15-30%. This is as a  result
of reading from multiple segments in the compaction pipeline. Applying merge on pipeline segments
more often should eliminate this overhead.

We experimented with basic policy which merge segments in the pipeline upon each in-memory
flush, this indeed solved the problem.
We are currently working on bringing merge back to life including solving some bugs we identified
and adding test to cover this path.

> In-Memory MemStore Policy for Flattening and Compactions
> --------------------------------------------------------
>                 Key: HBASE-16417
>                 URL: https://issues.apache.org/jira/browse/HBASE-16417
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Anastasia Braginsky
>            Assignee: Eshcar Hillel
>             Fix For: 2.0.0
>         Attachments: HBASE-16417-benchmarkresults-20161101.pdf, HBASE-16417-benchmarkresults-20161110.pdf,
HBASE-16417-benchmarkresults-20161123.pdf, HBASE-16417-benchmarkresults-20161205.pdf, HBASE-16417-benchmarkresults-20170309.pdf

This message was sent by Atlassian JIRA

View raw message