hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11351) Experiments with OffHeap memstores
Date Sun, 15 Jun 2014 03:53:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031775#comment-14031775
] 

Lars Hofhansl commented on HBASE-11351:
---------------------------------------

Next:
* Make it work in 0.96+ (protobufs). Although could keep the current serialization as well
* Better serialization (that makes use of the the sorted nature of the keys and does prefix
compression)
* Only store key portion in the key of the map
* Fork the set implementation so that we do not need to store KV in a map (which in offheap
means two copies of the KV)
* Do comparisons off heap! Needs more access into the BTree details (i.e. to the data before
it is deserialized onto the heap again)

Not sure how much time I'll get on those. If somebody wants to experiment I'm happy to collaborate.


> Experiments with OffHeap memstores
> ----------------------------------
>
>                 Key: HBASE-11351
>                 URL: https://issues.apache.org/jira/browse/HBASE-11351
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>         Attachments: 11351-0.94-v2.txt, 11351-0.94.txt
>
>
> I have been experimenting with offheap memstores using an offheap BTree implementation
of ConcurrentMap provided by MapDB (www.mapdb.org)
> So far I hooked this up in an experimental way with inefficient serialization (in the
sense that due to our special usage of the set api we need to back the set by a map, which
in the off heap case means we need to serialize and store the value twice. It is used as key
and value). But it works.
> It's quite a bit slower and due to need to deserialize upon scan it produces *more* short
term garbage.
> Let's use this issue to discuss.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message