incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benedict Elliott Smith <belliottsm...@datastax.com>
Subject Re: Cassandra slow on some reads
Date Fri, 14 Mar 2014 13:07:52 GMT
To add to this, if the 24G is the JVM limit, Cassandra will actually be
using even more than this for bloom filters etc. that are managed off-heap.
So the amount of page cache left is almost certainly inadequate.


On 14 March 2014 13:01, Andras Szerdahelyi <
andras.szerdahelyi@ignitionone.com> wrote:

>  Is row cache enabled on this CF? Try disabling it. Seems like you might
> have a very wide row there.
>
>  Can you grep for GCInspector in your Cassandra log? 24G might be a bit
> too much for the Cassandra JVM, bogging down GC, and not leaving much to
> page cache ( 32G -24G - Tomcat ). I don't quite understand your reasoning
> here:
> > (I know that there is a lot of heap but I also have write heavy tasks
> and I want them to get into mem fast) .
>
>  So, I would try with default cassandra-env.sh JVM params too
>
>   From: Batranut Bogdan <batranub@yahoo.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>,
> Batranut Bogdan <batranub@yahoo.com>
> Date: Friday 14 March 2014 13:50
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Subject: Cassandra slow on some reads
>
>   Hello all,
>
>  Here is the environment:
>
>  I have a 6 node Cassandra cluster. On each node I have:
> - 32 G RAM
> - 24 G RAM for cassa
> - ~150 - 200 MB/s disk speed
> - tomcat 6 with axis2 webservice that uses the datastax java driver to make
> asynch reads / writes
> - replication factor for the keyspace is 3
>
> (I know that there is a lot of heap but I also have write heavy tasks and
> I want them to get into mem fast) .
>
> All nodes in the same data center
> The clients that read / write are in the same datacenter so network is
> Gigabit.
>
>  The table structure is like this: PK(key String , timestam int, column1
> string, col2 string) , list1 , list 2, list 3 .
>  There are about 300 milions individual keys.
>  There are about 100 timestamps for each key now, so the rows will get
> wider as time passes.
>
>  I am using datastax java driver to query the cluster.
>
>  I have ~450 queries that are like this: SELECT * FROM table where key =
> 'some string' and ts = some value; some value is close to present time.
>
>  The problem:
>
>  About 10 - 20 % of these queries take more than 5 seconds to execute, in
> fact, the majority of those take around 10 seconds.
>  When investigating I saw that if I have a slow response and I redo the
> query it will finish in 8 - 10 MILIseconds like the rest of the queries
> that I have.
>  I could not see using JConsole any spikes in CPU / memory when executing
> the queries. The rise in resource consumtion is very small on all nodes on
> the cluster. I expect such delays to be generated by a BIG increase in
> resource consumption.
>
>  Any comments will be appreciated.
>
>  Thank you.
>
>
>
>
>

Mime
View raw message