hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3484) Replace memstore's ConcurrentSkipListMap with our own implementation
Date Tue, 17 Dec 2013 06:32:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850170#comment-13850170

Lars Hofhansl commented on HBASE-3484:

>From [~mcorgan]...
bq. I've been pondering how to better compact the data in the memstore. Sometimes we see a
100MB memstore flush that is really 10MB of KeyValues, which gzips to like 2MB, meaning there
is a ton of pointer overhead.

This should better now. In various patches I removed:
* caching of the row key (HBASE-7279)
* caching of the timestamp (HBASE-7279)
* caching of the KV length (HBASE-9956)

That saves 12 bytes + sizeOf(rowKey) for each KeyValue in the memstore.
The KV in memory overhead now is: 56 bytes. (the memstoreTS is also stored in the HFiles).

> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>                 Key: HBASE-3484
>                 URL: https://issues.apache.org/jira/browse/HBASE-3484
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance
>    Affects Versions: 0.92.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>         Attachments: WIP_HBASE-3484.patch, hierarchical-map.txt, memstore_drag.png
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for
our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save
one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we
should be OK with licenses.

This message was sent by Atlassian JIRA

View raw message