incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Batranut Bogdan <batra...@yahoo.com>
Subject Re:
Date Thu, 13 Mar 2014 13:48:20 GMT
Thanks ,

Edward and David,
Your contribution lead me to the conclusion. Unknown to me, the partition key had 1 value.
So of course all info was stored on a single node. Having replication factor 3 lead to having
99 % CPU on 3 machines. 
Regarding RAM we have so much CPU power that I actualy run a tomcat server on each node and
tomcat prepares data for queries, sends them, retrieves results, processes them and sends
responses to clients. We are running 6 Intel® Xeon® E3-1270 v3    Quad-Core Haswell    Hyper-Threading.
So there is CPU power there to handle high java heap IMO.

Thank you all for your prompt responses.



On Thursday, March 13, 2014 1:38 AM, David McNelis <dmcnelis@gmail.com> wrote:
 
Not knowing anything about your data structure (to expand on what Edward said), you could
be running into something where you've got some hot keys that are getting the majority of
writes during those heavily loads.... more specifically I might look for a single key that
you're writing, since you're RF=3 and you have 3 nodes specifically that are causing problems.




On Wed, Mar 12, 2014 at 7:28 PM, Russ Bradberry <rbradberry@gmail.com> wrote:

I wouldn't go above 8G unless you have a very powerful machine that can keep the GC pauses
low.
>
>Sent from my iPhone
>
>On Mar 12, 2014, at 7:11 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
>
>
>That is too much ram for cassandra make that 6g to 10g. 
>>
>>The uneven perf could be because your requests do not shard evenly.
>>
>>On Wednesday, March 12, 2014, Batranut Bogdan <batranub@yahoo.com> wrote:
>>> Hello all,
>>>
>>> The environment:
>>>
>>> I have a 6 node Cassandra cluster. On each node I have:
>>> - 32 G RAM
>>> - 24 G RAM for cassa
>>> - ~150 - 200 MB/s disk speed
>>> - tomcat 6 with axis2 webservice that uses the datastax java driver to make
>>> asynch reads / writes 
>>> - replication factor for the keyspace is 3
>>>
>>> All nodes in the same data center 
>>> The clients that read / write are in the same datacenter so network is
>>> Gigabit.
>>>
>>> Writes are performed via exposed methods from Axis2 WS . The Cassandra Java
>>> driver uses the round robin load balancing policy so all the nodes in the
>>> cluster should be hit with write requests under heavy write or read load
>>> from multiple clients.
>>>
>>> I am monitoring all nodes with JConsole from another box.
>>>
>>> The problem:
>>>
>>> When wrinting to a particular column family, only 3 nodes have high CPU load
>>> ~ 80 - 99 %. The remaining 3 are at ~2 - 10 % CPU. During writes, reads
>>> timeout. 
>>>
>>> I need more speed for both writes of reads. Due to the fact that 3 nodes
>>> barely have CPU activity leads me to think that the whole potential for C*
>>> is not touched.
>>>
>>> I am running out of ideas...
>>>
>>> If further details about the environment I can provide them.
>>>
>>>
>>> Thank you very much.
>>
>>-- 
>>Sorry this was sent from mobile. Will do less grammar and spell check than usual.
>>
Mime
View raw message