hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Mon, 28 Mar 2016 16:40:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214437#comment-15214437

stack commented on HBASE-14921:

bq. As you have mentioned, CellBlock is implementing ConcurrentNavigbleMap because delegatee
of the CellSet is of type ConcurrentNavigbleMap and we want just to give CellSet another delegatee
instead of ConcurrentSkipListMap.

Please say more on this. CellSet is a NavigableMap (not a ConcurrentNavigableMap) so I'm missing
where we need the 'Concurrent' (is it in this patch?)

Your new names are better. I considered 'flat' Map but shied away given its meaning over in
spark/scala; I think it will be ok as long as you stick why its a 'flat' map in the javadoc
on CellFlatMap.

bq. ... it is Cells per chunk and it should be constant for all Chunks in a CellBlockSerialized,
because MSLAB always allocates Chunks with same size

How do you see this working? We do not control the size of inbound Cells. They could have
some regularity and they could also be erratic to the extreme (What to do when a 1G cell arrives
into a column family that up to this has been taking on metrics?)

I still do not see how the 3 * int is BYTES_IN_CELL. Not important.

bq. But we are not sure what are the configurations where ChunkPool is off on purpose and
what we can do with such configurations. Do you know?

It was introduced and off by default as is usual when new features. But as also happens this
is our practice, the facility was 'forgotten'. It came up then when our Lars noticed it and
wanted to remove it since it was not being used. It came up again recently in HBASE-15513

It would seem to make sense enabling it by default if we come up w/ a proper sizing. Having
it on seems to mess w/ G1GC too. Would need to figure that.

We need to do up a memory management doc. Between your work on Segments, Segment pipelines,
MSLAB chunks, chunk pools and bytebufferpools to host requests read from sockets, bucket cache
and reference counting bucketcache bucket blocks at read time, it would be good if we had
a map so we could trace a Cell on its travels. 

> Memory optimizations
> --------------------
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch
> Memory optimizations including compressed format representation and offheap allocations

This message was sent by Atlassian JIRA

View raw message