cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5863) In process (uncompressed) page cache
Date Mon, 25 Apr 2016 23:07:13 GMT


Pavel Yaskevich commented on CASSANDRA-5863:

bq. The key itself is a small and fixed part of the overhead (all objects it references are
already found elsewhere); there are also on-heap support structures within the implementing
cache which are bigger. Though that's not trivial, we could also account for those, but I
don't know how that helps cache management and sizing for the user.

The problem I see with this is the same for any other data structure on JVM - if we don't
account for additional overhead at some point it will blow up and it won't be pretty, especially
if we don't account for internal size of the data structure which holds the cache and other
overhead like keys and their containers, can we claim with certainty that at some capacity
it's actual size in memory is not going to be 2x or 3x? If yes then let's leave it like it
is today otherwise we need to do something about it right away.

bq. I'm sorry, I do not understand the problem – the code only relies on the position of
the buffer and since buffer is cleared before the read, an end of stream (and only that) will
result in an empty buffer; both read() and readByte() interpret this correctly.

Sorry, what I mean - we might want to be more conservative and indicate early that requested
length is bigger than number of available bytes, we already had couple of bugs which where
hard to debug because EOFException doesn't provide any useful information...

bq. I had added a return of the passed buffer for convenience but it also adds possibility
for error – changed the return of the method to void. On the other point, it does not make
sense for the callee to return an (aligned) offset as the caller may need to have a better
control over positioning before allocating the buffer – caching rebufferers, specifically,


bq. This wasn't the case even before this ticket. When RAR requests rebuffering at a certain
position, it can either have its buffer filled (direct case), or receive a view of a shared
buffer that holds the data (mem-mapped case). There was a lot of clumsiness in RAR to handle
the question of which of these is the case, does it own its buffer, should it be allocated
or freed. The patch addresses this clumsiness as well as allowing for another type of advantageous
buffer management.

I understand, I actually started with proposition to return "void" but I changed it later
on because I saw a possibility to unify bufferless with other implementations because essentially
the question is - where original data comes from - directly from the channel or already mmap'ed
buffer, so maybe if we had a common interface to both of the cases and used it as a backend
for rebufferer it would simplify things instead of putting that logic into rebufferer itself?
Just something to think about...

bq. Interesting. Another possibility mentioned before is to implement compression in such
a way that the compressed size matches the chunk size. Both are orthogonal and outside the
scope of this ticket – lets open a new issue for that?

I'm fine if we make it a separate ticket but I think we will have to tackle it first since
it would directly affect rebufferer/cache logic.

> In process (uncompressed) page cache
> ------------------------------------
>                 Key: CASSANDRA-5863
>                 URL:
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: T Jake Luciani
>            Assignee: Branimir Lambov
>              Labels: performance
>             Fix For: 3.x
> Currently, for every read, the CRAR reads each compressed chunk into a byte[], sends
it to ICompressor, gets back another byte[] and verifies a checksum.  
> This process is where the majority of time is spent in a read request.  
> Before compression, we would have zero-copy of data and could respond directly from the
> It would be useful to have some kind of Chunk cache that could speed up this process
for hot data, possibly off heap.

This message was sent by Atlassian JIRA

View raw message