cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6689) Partially Off Heap Memtables
Date Mon, 03 Mar 2014 23:35:32 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918726#comment-13918726
] 

Benedict edited comment on CASSANDRA-6689 at 3/3/14 11:35 PM:
--------------------------------------------------------------

bq. can you elaborate why do we need to explicitly set it to null?

I do this in places where I'm handing off control of the referrer to another thread/context,
so we only call setDone() in the case of an error before we did this. Admittedly, looking
at these places now, it looks like replacing the finally block with a catch (Throwable t)
block would be fine, and given we can do untyped rethrow there isn't anything lost by doing
this. Probably an old habit from Java 6 to do it the way I did. I will fix this in the CASSANDRA-6694
branch.

bq. With would add more control in the situation when we flush memtables in short intervals.

If I understand you correctly, you mean to deal with the situation where we want to flush
memtables quick enough that the single cleaner thread may become the bottleneck? In which
case, I don't think this should ever be a problem: the work done by the cleaner itself is
very minimal unless it's got some GC to do, in which case it is potentially non-trivial, but
still unlikely to be a bottleneck. There might be a case for handing off the GC candidate
selection to another thread, except that we expect it to certainly be quicker than any flush,
and we want to wait for it to complete before deciding if flushing is actually worthwhile,
so I'm not sure it buys us much. Maybe we could do this and continue to check the memory state
isn't worsening, and if it is just force a flush without waiting for the GC, but it probably
is too unlikely to be of benefit to be worth the extra complexity. That is, unless you meant
some other benefit I missed?

bq. Related question, why can't we have per-cf (or global) pool instead of creating it per
memtable, might be possible to remove allocator groups if we do that?

The pool _is_ global; a per-cf pool would have the issue of poor sharing of resources, though,
no? I would probably like to see us move to allocators surviving across memtable replacement,
so that we could then flatten allocator group with allocator, but I'd prefer to save that
for a later date.

bq. Also I just want to throw an idea here maybe there is something useful in it - as we have
fixed number of stages and threads in them and don't normally share much between them, maybe
it would be possible to use that as an advantage for allocator gc? What I mean is, it might
be possible to track life-time of the allocation not by how buffer was allocated but where
it was allocated, more like what RCU...

Not really sure what you're suggesting here, could you elaborate?


was (Author: benedict):
bq. can you elaborate why do we need to explicitly set it to null?

I do this in places where I'm handing off control of the referrer to another thread/context,
so we only call setDone() in the case of an error before we did this. Admittedly, looking
at these places now, it looks like replacing the finally block with a catch (Throwable t)
block would be fine, and given we can do untyped rethrow there isn't anything lost by doing
this. Probably an old habit from Java 6 to do it the way I did.

bq. With would add more control in the situation when we flush memtables in short intervals.

If I understand you correctly, you mean to deal with the situation where we want to flush
memtables quick enough that the single cleaner thread may become the bottleneck? In which
case, I don't think this should ever be a problem: the work done by the cleaner itself is
very minimal unless it's got some GC to do, in which case it is potentially non-trivial, but
still unlikely to be a bottleneck. There might be a case for handing off the GC candidate
selection to another thread, except that we expect it to certainly be quicker than any flush,
and we want to wait for it to complete before deciding if flushing is actually worthwhile,
so I'm not sure it buys us much. Maybe we could do this and continue to check the memory state
isn't worsening, and if it is just force a flush without waiting for the GC, but it probably
is too unlikely to be of benefit to be worth the extra complexity. That is, unless you meant
some other benefit I missed?

bq. Related question, why can't we have per-cf (or global) pool instead of creating it per
memtable, might be possible to remove allocator groups if we do that?

The pool _is_ global; a per-cf pool would have the issue of poor sharing of resources, though,
no? I would probably like to see us move to allocators surviving across memtable replacement,
so that we could then flatten allocator group with allocator, but I'd prefer to save that
for a later date.

bq. Also I just want to throw an idea here maybe there is something useful in it - as we have
fixed number of stages and threads in them and don't normally share much between them, maybe
it would be possible to use that as an advantage for allocator gc? What I mean is, it might
be possible to track life-time of the allocation not by how buffer was allocated but where
it was allocated, more like what RCU...

Not really sure what you're suggesting here, could you elaborate?

> Partially Off Heap Memtables
> ----------------------------
>
>                 Key: CASSANDRA-6689
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6689
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Benedict
>            Assignee: Benedict
>             Fix For: 2.1 beta2
>
>
> Move the contents of ByteBuffers off-heap for records written to a memtable.
> (See comments for details)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message