incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Batranut Bogdan <batra...@yahoo.com>
Subject Cassandra slow on some reads
Date Fri, 14 Mar 2014 12:50:39 GMT
Hello all,

Here is the environment:

I have a 6 node Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RAM for cassa
- ~150 - 200 MB/s disk speed
- tomcat 6 with axis2 webservice that uses the datastax java driver to make
asynch reads / writes 
- replication factor for the keyspace is 3

(I know that there is a lot of heap but I also have write heavy tasks and I want them to get
into mem fast) .

All nodes in the same data center 
The clients that read / write are in the same datacenter so network is Gigabit.


The table structure is like this: PK(key String , timestam int, column1 string, col2 string)
, list1 , list 2, list 3 .
There are about 300 milions individual keys.
There are about 100 timestamps for each key now, so the rows will get wider as time passes.

I am using datastax java driver to query the cluster.

I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and
ts = some value; some value is close to present time.

The problem:

About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority
of those take around 10 seconds.
When investigating I saw that if I have a slow response and I redo the query it will finish
in 8 - 10 MILIseconds like the rest of the queries that I have. 
I could not see using JConsole any spikes in CPU / memory when executing the queries. The
rise in resource consumtion is very small on all nodes on the cluster. I expect such delays
to be generated by a BIG increase in resource consumption.

Any comments will be appreciated.

Thank you.
Mime
View raw message