cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6689) Partially Off Heap Memtables
Date Thu, 06 Mar 2014 22:09:44 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923147#comment-13923147
] 

Pavel Yaskevich edited comment on CASSANDRA-6689 at 3/6/14 10:08 PM:
---------------------------------------------------------------------

Doing everything together is even bigger opportunity to mess up. Let me clarify my point,
hopefully you will hear it this time as you plead for committers to decide so I'm representing.
From my side nothing from this ticket would go in until we separate all of the work into multiple
committable stages, namely (but not necessarily exactly like that): 

# Make use off-heap memory inside of Memtables, which delivers increased capacity and less
frequent/painful CMS (commit #1):
  a. Determine if we actually need to re-implement custom allocator or can go with existing
one e.g. netty's implementation
  b. plug allocator from #1 into Memtable for memory usage tracking, do all necessary bookkeeping
to release memory back to the pool once it's no longer needed (overwrite, flush etc.), make
Memtable memory life time limited by copying data from Memtable on the read path.
# Optimizations after #1 which determine how big of impact the copying actually makes comparing
to original situation on different workloads and how good is ref tracking comparing to that
(commit #2)
   Delivers - memory reference tracking (RefAction, GC + what ever is needed)

(if needed) # Further enhancements to allocators/ref accounting (commit #3)

This way we can reuse already existing RefAction and related code that we already have, commit
#1 doesn't require any substantial changes in the way we treat memtables and adds decoupled
allocator framework, commit #2 plugs reference tracking into all previous changes, commit
#3 further improves things if needed.


   


was (Author: xedin):
Doing everything together is even bigger opportunity to mess up. Let me clarify my point,
hopefully you will hear it this time as you plead for committers to decide so I'm representing.
From my side nothing from this ticket would go in until we separate all of the work into multiple
committable stages, namely (but not necessarily exactly like that): 

# Make use off-heap memory inside of Memtables, which delivers increased capacity and less
frequent/painful CMS (commit #1):
  a. Determine if we actually need to re-implement custom allocator or can go with existing
one e.g. netty's implementation
  b. plug allocator from #1 into Memtable for memory usage tracking, do all necessary bookkeeping
to release memory back to the pool once it's no longer needed (overwrite, flush etc.), make
Memtable memory life time limited by copying data from Memtable on the read path.

# Optimizations after #1 which determine how big of impact the copying actually makes comparing
to original situation on different workloads and how good is ref tracking comparing to that
(commit #2)
   Delivers - memory reference tracking (RefAction, GC + what ever is needed)

(if needed) # Further enhancements to allocators/ref accounting (commit #3)

This way we can reuse already existing RefAction and related code that we already have, commit
#1 doesn't require any substantial changes in the way we treat memtables and adds decoupled
allocator framework, commit #2 plugs reference tracking into all previous changes, commit
#3 further improves things if needed.


   

> Partially Off Heap Memtables
> ----------------------------
>
>                 Key: CASSANDRA-6689
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6689
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Benedict
>            Assignee: Benedict
>             Fix For: 2.1 beta2
>
>         Attachments: CASSANDRA-6689-small-changes.patch
>
>
> Move the contents of ByteBuffers off-heap for records written to a memtable.
> (See comments for details)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message