cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Stevens <migh...@gmail.com>
Subject Re: High GC activity on node with 4TB on data
Date Thu, 12 Feb 2015 15:18:36 GMT
> each node has 256G of memory, 24x1T drives, 2x Xeon CPU

I don't have first hand experience running Cassandra on such massive
hardware, but it strikes me that these machines are dramatically oversized
to be good candidates for Cassandra (though I wonder how many cores are in
those CPUs; I'm guessing closer to 18 than 2 based on the other hardware).

A larger cluster of smaller hardware would be a much better shape for
Cassandra.  Or several clusters of smaller hardware since you're running
multiple instances on this hardware - best practices have one instance per
host no matter the hardware size.

On Thu, Feb 12, 2015 at 12:36 AM, Jiri Horky <horky@avast.com> wrote:

>  Hi Chris,
>
> On 02/09/2015 04:22 PM, Chris Lohfink wrote:
>
>   - number of tombstones - how can I reliably find it out?
>  https://github.com/spotify/cassandra-opstools
>  https://github.com/cloudian/support-tools
>
> thanks.
>
>
>  If not getting much compression it may be worth trying to disable it, it
> may contribute but its very unlikely that its the cause of the gc pressure
> itself.
>
>  7000 sstables but STCS? Sounds like compactions couldn't keep up.  Do
> you have a lot of pending compactions (nodetool)?  You may want to increase
> your compaction throughput (nodetool) to see if you can catch up a little,
> it would cause a lot of heap overhead to do reads with that many.  May even
> need to take more drastic measures if it cant catch back up.
>
> I am sorry, I was wrong. We actually do use LCS (the switch was done
> recently). There are almost none pending compaction. We have increased the
> size sstable to 768M, so it should help as as well.
>
>
>  May also be good to check `nodetool cfstats` for very wide partitions.
>
> There are basically none, this is fine.
>
> It seems that the problem really comes from having so much data in so many
> sstables, so org.apache.cassandra.io.compress.CompressedRandomAccessReader
> classes consumes more memory than 0.75*HEAP_SIZE, which triggers the CMS
> over and over.
>
> We have turned off the compression and so far, the situation seems to be
> fine.
>
> Cheers
> Jirka H.
>
>
>
>  Theres a good chance if under load and you have over 8gb heap your GCs
> could use tuning.  The bigger the nodes the more manual tweaking it will
> require to get the most out of them
> https://issues.apache.org/jira/browse/CASSANDRA-8150 also has some ideas.
>
>  Chris
>
> On Mon, Feb 9, 2015 at 2:00 AM, Jiri Horky <horky@avast.com> wrote:
>
>>  Hi all,
>>
>> thank you all for the info.
>>
>> To answer the questions:
>>  - we have 2 DCs with 5 nodes in each, each node has 256G of memory,
>> 24x1T drives, 2x Xeon CPU - there are multiple cassandra instances running
>> for different project. The node itself is powerful enough.
>>  - there 2 keyspaces, one with 3 replicas per DC, one with 1 replica per
>> DC (because of amount of data and because it serves more or less like a
>> cache)
>>  - there are about 4k/s Request-response, 3k/s Read and 2k/s Mutation
>> requests  - numbers are sum of all nodes
>>  - we us STCS (LCS would be quite IO have for this amount of data)
>>  - number of tombstones - how can I reliably find it out?
>>  - the biggest CF (3.6T per node) has 7000 sstables
>>
>> Now, I understand that the best practice for Cassandra is to run "with
>> the minimum size of heap which is enough" which for this case we thought is
>> about 12G - there is always 8G consumbed by the SSTable readers. Also, I
>> though that high number of tombstones create pressure in the new space
>> (which can then cause pressure in old space as well), but this is not what
>> we are seeing. We see continuous GC activity in Old generation only.
>>
>> Also, I noticed that the biggest CF has Compression factor of 0.99 which
>> basically means that the data come compressed already. Do you think that
>> turning off the compression should help with memory consumption?
>>
>> Also, I think that tuning CMSInitiatingOccupancyFraction=75 might help
>> here, as it seems that 8G is something that Cassandra needs for bookkeeping
>> this amount of data and that this was sligtly above the 75% limit which
>> triggered the CMS again and again.
>>
>> I will definitely have a look at the presentation.
>>
>> Regards
>> Jiri Horky
>>
>>
>> On 02/08/2015 10:32 PM, Mark Reddy wrote:
>>
>> Hey Jiri,
>>
>>  While I don't have any experience running 4TB nodes (yet), I would
>> recommend taking a look at a presentation by Arron Morton on large nodes:
>> http://planetcassandra.org/blog/cassandra-community-webinar-videoslides-large-nodes-with-cassandra-by-aaron-morton/
>> to see if you can glean anything from that.
>>
>>  I would note that at the start of his talk he mentions that in version
>> 1.2 we can now talk about nodes around 1 - 3 TB in size, so if you are
>> storing anything more than that you are getting into very specialised use
>> cases.
>>
>>  If you could provide us with some more information about your cluster
>> setup (No. of CFs, read/write patterns, do you delete / update often, etc.)
>> that may help in getting you to a better place.
>>
>>
>>  Regards,
>> Mark
>>
>> On 8 February 2015 at 21:10, Kevin Burton <burton@spinn3r.com> wrote:
>>
>>> Do you have a lot of individual tables?  Or lots of small compactions?
>>>
>>>  I think the general consensus is that (at least for Cassandra), 8GB
>>> heaps are ideal.
>>>
>>>  If you have lots of small tables it’s a known anti-pattern (I believe)
>>> because the Cassandra internals could do a better job on handling the in
>>> memory metadata representation.
>>>
>>>  I think this has been improved in 2.0 and 2.1 though so the fact that
>>> you’re on 1.2.18 could exasperate the issue.  You might want to consider an
>>> upgrade (though that has its own issues as well).
>>>
>>> On Sun, Feb 8, 2015 at 12:44 PM, Jiri Horky <horky@avast.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> we are seeing quite high GC pressure (in old space by CMS GC Algorithm)
>>>> on a node with 4TB of data. It runs C* 1.2.18 with 12G of heap memory
>>>> (2G for new space). The node runs fine for couple of days when the GC
>>>> activity starts to raise and reaches about 15% of the C* activity which
>>>> causes dropped messages and other problems.
>>>>
>>>> Taking a look at heap dump, there is about 8G used by SSTableReader
>>>> classes in
>>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.
>>>>
>>>> Is this something expected and we have just reached the limit of how
>>>> many data a single Cassandra instance can handle or it is possible to
>>>> tune it better?
>>>>
>>>> Regards
>>>> Jiri Horky
>>>>
>>>
>>>
>>>
>>>   --
>>>   Founder/CEO Spinn3r.com
>>>  Location: *San Francisco, CA*
>>>  blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>>  <http://spinn3r.com>
>>>
>>
>>
>>
>
>

Mime
View raw message