directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christoph Engelbert <noctar...@apache.org>
Subject Re: New buffer backend
Date Sun, 21 Jul 2013 11:39:18 GMT
Hey guys
What I forgot to mention is that we definately have to explain the
parameters when creating the MemoryManager since for the new buffer
backend they are different to the original ones. I use the
concurrencylevel which is already used by guava to decide on the
partition count and use the buffercount as the count of slices
inside of an partition.

A bit more on the internals of the new buffer framework:

There are a number of partitions which are sliced into multiple
fragments, all of the same size (a divisor of the partition size).
Partition 1 (fragment size for example 128 bytes)
+------------------------------------------ ----------+
|  1  |  2  |  3  |  4  |  5  |  6  |  7  |  8  |  9  |
+-----------------------------------------------------+
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
+-----------------------------------------------------+
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
+-----------------------------------------------------+

Partition 2
+------------------------------------------ ----------+
|  1  |  2  |  3  |  4  |  5  |  6  |  7  |  8  |  9  |
+-----------------------------------------------------+
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
+-----------------------------------------------------+
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
+-----------------------------------------------------+

If you request a new PartitionBuffer you have the option to give it
a base size, if not given a single fragment will be requested.

PartitionBufferPool.getPartitionBuffer() => ParitionBuffer1 [[P1,1]]
PartitionBufferPool.getPartitionBuffer(256) => ParitionBuffer2
[[P2,1], [P2,2]]

All PartitionBuffers are auto growing as long as there are free
fragments available so if you try to write beyond the current max
capacity in behind the buffer requests a new fragment to write to:

PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,1]]

Fragments don't to be received from the same partition, this heavily
depends on the choosing selection strategy (like cpu local, thread
local, round robin) and if the current partition has some more free
fragments. Depending on the selection strategy different best
practices for partition count are available.

What is important to note for the new buffer backend, the memory
usage is higher for small pieces of data or datasize little bit
bigger than multiple of fragment size. So in example you'll waste
lots of memory if you would store values of 129 bytes (2 fragments
with second fragment only used for one byte). This is a part that is
best for user to configure correctly but the most important part is,
this is totally different behavior than for old MemoryManagers which
sliced to the correct size.

I guess that's it so far, any questions? I may would appreciate a
hangout or something like that to do some further explanation and to
get opinions, ideas, review.

Chris

Am 17.07.2013 19:29, schrieb Raffaele P. Guidi:
> Great ! :)
> Il giorno 17/lug/2013 12:41, "Christoph Engelbert" <noctarius@apache.org>
> ha scritto:

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message