incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Digest Query Seems to be corrupt on certain cases
Date Sun, 31 Mar 2013 09:37:00 GMT
> When I manually inspected this byte array, it seems hold all details correctly, except
the super-column name, causing it to fetch the entire wide row.
What is the CF definition and what is the exact query you are sending? 
There does not appear to be anything obvious in the QueryPath serde for 1.0.7

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 28/03/2013, at 10:54 AM, Ravikumar Govindarajan <ravikumar.govindarajan@gmail.com>
wrote:

> VM Settings are
> -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
-Xms8G -Xmx8G -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
> 
> error stack was containing 2 threads for the same key, stalling on digest query
> 
> The below bytes which I referred is the actual value of "_body" variable in org.apache.cassandra.net.Message
object got from the heap dump.
> 
> As I understand from the code, ReadVerbHandler will deserialize this "_body" variable
into a SliceByNamesReadCommand object.
> 
> When I manually inspected this byte array, it seems hold all details correctly, except
the super-column name, causing it to fetch the entire wide row.
> 
> --
> Ravi
> 
> On Thu, Mar 28, 2013 at 8:36 AM, aaron morton <aaron@thelastpickle.com> wrote:
>> We started receiving OOMs in our cassandra grid and took a heap dump
> What are the JVM settings ? 
> What was the error stack? 
> 
>> I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to
be corrupt on issuing certain digest queries.
> 
> Sorry I don't follow what you are saying here. 
> Can you can you enable DEBUG logging and identify the behaviour you think is incorrect
?
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan <ravikumar.govindarajan@gmail.com>
wrote:
> 
>> We started receiving OOMs in our cassandra grid and took a heap dump. We are running
version 1.0.7 with LOCAL_QUORUM from both reads/writes.
>> 
>> After some analysis, we kind of identified the problem, with SliceByNamesReadCommand,
involving a single Super-Column. This seems to be happening only in digest query and not during
actual reads.
>> 
>> I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to
be corrupt on issuing certain digest queries.
>> 
>> 		//Type is SliceByNamesReadCommand
>> 		body[0] = (byte)1;
>> 		
>> 		//This is a digest query here.
>> 		body[1] = (byte)1;
>> 
>>                 //Table-Name from 2-8 bytes
>> 
>>                 //Key-Name from 9-18 bytes
>> 
>>                 //QueryPath deserialization here
>>              
>>                      //CF-Name from 19-30 bytes
>> 
>>                     //Super-Col-Name from 31st byte onwards, but gets corrupt as
found in heap dump
>> 
>>                     //body[32-37] = 0, body[38] = 1, body[39] = 0.  This causes the
SliceByNamesDeserializer to mark both ColName=NULL and SuperColName=NULL, fetching entire
wide-row!!!
>> 
>>                    //Actual super-col-name starts only from byte 40, whereas it should
have started from 31st byte itself
>> 
>> Has someone already encountered such an issue? Why is the super-col-name not correctly
de-serialized during digest query.
>> 
>> --
>> Ravi
>> 
> 
> 


Mime
View raw message