hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Corgan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3484) Replace memstore's ConcurrentSkipListMap with our own implementation
Date Tue, 10 Jul 2012 21:26:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410934#comment-13410934
] 

Matt Corgan commented on HBASE-3484:
------------------------------------

I've been pondering how to better compact the data in the memstore.  Sometimes we see a 100MB
memstore flush that is really 10MB of KeyValues, which gzips to like 2MB, meaning there is
a ton of pointer overhead.

One thing that came to mind was splitting each memstore into "regions" of consecutive cell
ranges and fronting these regions with an index of some sort.  Instead of Set<KeyValue>
the memstore is Set<Set<KeyValue>>.  When an internal region crosses a certain
size we split it in half.  With a good index structure in front of the memstore blocks, it
might get closer to a linear performance/size curve.  It's comparable with hbase splitting
a table into regions.

Then, to address the pointer overhead problem, you could use DataBlockEncoding to encode each
memstore region individually.  A memstore region could accumulate several blocks that get
compacted periodically.  Given a region size of ~64-256KB, the compaction could be very aggressive
and could even be done by the thread writing the data.  Again, very similar to how hbase manages
the internals of a single region.

This adds moving pieces and complexity but could be developed as a pluggable module that passes
the same unit tests as the current memstore.
                
> Replace memstore's ConcurrentSkipListMap with our own implementation
> --------------------------------------------------------------------
>
>                 Key: HBASE-3484
>                 URL: https://issues.apache.org/jira/browse/HBASE-3484
>             Project: HBase
>          Issue Type: Improvement
>          Components: performance
>    Affects Versions: 0.92.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>         Attachments: hierarchical-map.txt, memstore_drag.png
>
>
> By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements to it for
our use case in MemStore:
> - add an iterator.replace() method which should allow us to do upsert much more cheaply
> - implement a Set directly without having to do Map<KeyValue,KeyValue> to save
one reference per entry
> It turns out CSLM is in public domain from its development as part of JSR 166, so we
should be OK with licenses.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message