incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Batranut Bogdan <batra...@yahoo.com>
Subject Re: Cassandra slow on some reads
Date Fri, 14 Mar 2014 14:35:51 GMT
Well the problem still persists.

Giving cassandra 12G of heap and having a look at the table I saw that  caching='KEYS_ONLY'
. Did not find how to disable caching for rows (I'm not sure if setting to 0 will disable
it)





On Friday, March 14, 2014 3:14 PM, Andras Szerdahelyi <andras.szerdahelyi@ignitionone.com>
wrote:
 
Is row cache enabled on this CF? Try disabling it. Seems like you might have a very wide row
there.

Can you grep for GCInspector in your Cassandra log? 24G might be a bit too much for the Cassandra
JVM, bogging down GC, and not leaving much to page cache ( 32G -24G - Tomcat ). I don’t
quite understand your reasoning here:
> (I know that there is a lot of heap but I also have write heavy tasks and I want them
to get into mem fast) .

So, I would try with default cassandra-env.sh JVM params too

From: Batranut Bogdan <batranub@yahoo.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>, Batranut Bogdan <batranub@yahoo.com>
Date: Friday 14 March 2014 13:50
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Cassandra slow on some reads


Hello all,

Here is the environment:

I have a 6 node Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RAM for cassa
- ~150 - 200 MB/s disk speed
- tomcat 6 with axis2 webservice that uses the datastax java driver to make
asynch reads / writes 
- replication factor for the keyspace is 3

(I know that there is a lot of heap but I also have write heavy tasks and I want them to get
into mem fast) .

All nodes in the same data center 
The clients that read / write are in the same datacenter so network is Gigabit.


The table structure is like this: PK(key String , timestam int, column1 string, col2 string)
, list1 , list 2, list 3 .
There are about 300 milions individual keys.
There are about 100 timestamps for each key now, so the rows will get wider as time passes.

I am using datastax java driver to query the cluster.

I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and
ts = some value; some value is close to present time.

The problem:

About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority
of those take around 10 seconds.
When investigating I saw that if I have a slow response and I redo the query it will finish
in 8 - 10 MILIseconds like the rest of the queries that I have. 
I could not see using JConsole any spikes in CPU / memory when executing the queries. The
rise in resource consumtion is very small on all nodes on the cluster. I expect such delays
to be generated by a BIG increase in resource consumption.

Any comments will be appreciated.

Thank you.
Mime
View raw message