incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravikumar Govindarajan <ravikumar.govindara...@gmail.com>
Subject Re: Digest Query Seems to be corrupt on certain cases
Date Thu, 28 Mar 2013 05:24:13 GMT
VM Settings are
-javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn800M
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

error stack was containing 2 threads for the same key, stalling on digest
query

The below bytes which I referred is the actual value of "_body" variable in
org.apache.cassandra.net.Message object got from the heap dump.

As I understand from the code, ReadVerbHandler will deserialize this
"_body" variable into a SliceByNamesReadCommand object.

When I manually inspected this byte array, it seems hold all details
correctly, except the super-column name, causing it to fetch the entire
wide row.

--
Ravi

On Thu, Mar 28, 2013 at 8:36 AM, aaron morton <aaron@thelastpickle.com>wrote:

> We started receiving OOMs in our cassandra grid and took a heap dump
>
> What are the JVM settings ?
> What was the error stack?
>
> I am pasting the serialized byte array of SliceByNamesReadCommand, which
> seems to be corrupt on issuing certain digest queries.
>
> Sorry I don't follow what you are saying here.
> Can you can you enable DEBUG logging and identify the behaviour you think
> is incorrect ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan <
> ravikumar.govindarajan@gmail.com> wrote:
>
> We started receiving OOMs in our cassandra grid and took a heap dump. We
> are running version 1.0.7 with LOCAL_QUORUM from both reads/writes.
>
> After some analysis, we kind of identified the problem, with
> SliceByNamesReadCommand, involving a single Super-Column. This seems to be
> happening only in digest query and not during actual reads.
>
> I am pasting the serialized byte array of SliceByNamesReadCommand, which
> seems to be corrupt on issuing certain digest queries.
>
> //Type is SliceByNamesReadCommand
>  body[0] = (byte)1;
>  //This is a digest query here.
>  body[1] = (byte)1;
>
>                 //Table-Name from 2-8 bytes
>
>                 //Key-Name from 9-18 bytes
>
>                 //QueryPath deserialization here
>
>                      //CF-Name from 19-30 bytes
>
>                     //Super-Col-Name from 31st byte onwards, but gets
> corrupt as found in heap dump
>
>                     //body[32-37] = 0, body[38] = 1, body[39] = 0.  This
> causes the SliceByNamesDeserializer to mark both ColName=NULL and
> SuperColName=NULL, fetching entire wide-row!!!
>
>                    //Actual super-col-name starts only from byte 40,
> whereas it should have started from 31st byte itself
>
> Has someone already encountered such an issue? Why is the super-col-name
> not correctly de-serialized during digest query.
>
> --
> Ravi
>
>
>

Mime
View raw message