cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Szerdahelyi <>
Subject Re: Cassandra slow on some reads
Date Fri, 14 Mar 2014 13:01:03 GMT
Is row cache enabled on this CF? Try disabling it. Seems like you might have a very wide row

Can you grep for GCInspector in your Cassandra log? 24G might be a bit too much for the Cassandra
JVM, bogging down GC, and not leaving much to page cache ( 32G -24G - Tomcat ). I don’t
quite understand your reasoning here:
> (I know that there is a lot of heap but I also have write heavy tasks and I want them
to get into mem fast) .

So, I would try with default JVM params too

From: Batranut Bogdan <<>>
Reply-To: "<>" <<>>,
Batranut Bogdan <<>>
Date: Friday 14 March 2014 13:50
To: "<>" <<>>
Subject: Cassandra slow on some reads

Hello all,

Here is the environment:

I have a 6 node Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RAM for cassa
- ~150 - 200 MB/s disk speed
- tomcat 6 with axis2 webservice that uses the datastax java driver to make
asynch reads / writes
- replication factor for the keyspace is 3

(I know that there is a lot of heap but I also have write heavy tasks and I want them to get
into mem fast) .

All nodes in the same data center
The clients that read / write are in the same datacenter so network is Gigabit.

The table structure is like this: PK(key String , timestam int, column1 string, col2 string)
, list1 , list 2, list 3 .
There are about 300 milions individual keys.
There are about 100 timestamps for each key now, so the rows will get wider as time passes.

I am using datastax java driver to query the cluster.

I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and
ts = some value; some value is close to present time.

The problem:

About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority
of those take around 10 seconds.
When investigating I saw that if I have a slow response and I redo the query it will finish
in 8 - 10 MILIseconds like the rest of the queries that I have.
I could not see using JConsole any spikes in CPU / memory when executing the queries. The
rise in resource consumtion is very small on all nodes on the cluster. I expect such delays
to be generated by a BIG increase in resource consumption.

Any comments will be appreciated.

Thank you.

View raw message