directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Kotek <dis...@kotek.net>
Subject Re: New buffer backend
Date Mon, 22 Jul 2013 20:28:09 GMT
Hi,

I have something similar in MapDB. It is in Volume.java, a light growable 
abstraction ByteBuffers and  FileChannel. Major difference is that Volume is 
not thread safe, locking is handled at higher level. It will be interesting to 
compare those. 

Any chance this backed would work for memory mapped files?

Jan Kotek



On Friday 12 July 2013 20:55:11 Christoph Engelbert wrote:
> Hey guys
> 
> I finally managed to merge everything together :-)
> 
> As stated a few weeks before I made a partitioned buffer system for
> good performance and less contention.
> It had different selection strategies like TLA (Thread Local
> Allocation), a simple RoundRobin or (on Linux and Windows) CLA
> (Processor Core Local Allocation), whereas the last is done using OS
> calls and JNA.
> 
> It features ByteBuffers for Heap and Offheap as well as Unsafe. It
> has growing buffers (if slice is full a new one is selected) and can
> handle data bigger than Integer.MAX_VALUE (it uses full long
> position pointers).
> 
> It is located in directmemory-buffer submodule since it was it's own
> project and it is fully usable even without having to use
> DirectMemory (I would suggest to give users the chance to use it on
> their own).
> 
> As stated before it introduces a new dependency and especially a
> platform depending one. At least it is a optional dependency and CLA
> is deactivated if JNA is not available on the classpath.
> 
> I although added 3 properties to configure the default strategy of
> creating the PartitionBufferPools:
> directmemory.buffer.pooling.disabled: true deactivates pooling and
> uses lazy creation and immediate destroying on release
> directmemory.buffer.unsafe.enabled: true activates the usage of
> sun.misc.Unsafe raw memory access (a check if unsafe is available is
> applied too)
> directmemory.buffer.offheap.enabled: true enabled DirectByteBuffer
> usage for for the non-unsafe-pools
> 
> I merged it into my local fork of DirectMemory on github [1] but had
> to adjust the API of DirectMemory on some places. I introduced a
> MemoryManagerFactory which handles creation of the different
> MemoryManagers (as the old ones - partly renamed -
> UnsafeMemoryManager and AllocatorMemoryManager) and the new
> PartitionBufferMemoryManager.
> The Pointer-API is now able to use PartitionBuffers as well as the
> old way using byte[].
> 
> I'm not yet finished, working on making all unittests pass again but
> I would appreciate some opinions, discussion on the new API changes.
> 
> Cheers
> Chris
> 
> [1] https://github.com/noctarius/directmemory/tree/buffer
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message