cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: node re-start delays , busy Deleting mc-txn-compaction/ Adding log file replica
Date Thu, 20 Jun 2019 10:25:37 GMT
Hello Asad,


> I’m on environment with  apache Cassandra 3.11.1 with  java 1.8.0_144.

One Node went OOM and crashed.


If I remember well, firsts minor versions of C* 3.11 have memory leaks. It
seems it was fixed in your version though.

3.11.1

[...]

 * BTree.Builder memory leak (CASSANDRA-13754)


Yet other improvements were made later on:


> 3.11.3

[...]

 * Remove BTree.Builder Recycler to reduce memory usage (CASSANDRA-13929)

 * Reduce nodetool GC thread count (CASSANDRA-14475)


See: https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt.
Before digging more I would upgrade to 3.11.latest (latest = 4 or 5 I
guess), because early versions of a major Cassandra versions are famous for
being quite broken, even though this major is a 'bug fix only' branch.
Also minor versions upgrades are not too risky to go through. I would maybe
start there if you're not too sure how to dig this.

If it happens again or you don't want to upgrade, it would be interesting
to know:
-  if the OOM happens inside the JVM or on native memory (then the OS would
be the one sending the kill signal). These 2 issues have different (and
sometime opposite) fixes.
- What's the host size (especially memory) and how the heap (and maybe some
off heap structures) are configured (at least what is not default).
- If you saw errors in the logs and what the 'nodetool tpstats' was looking
like when the node went down (it might have been dumped in the logs)

I don't know much about those traces nor why Cassandra would take a long
time. Though they are traces and harder to interpret for me. What does the
INFO / WARN / ERR look like?
Maybe opening a lot of SSTables and/or replaying a lot of commit logs,
given the nature of the restart (post outage)?
To speed up things, when nodes are not crashing, under normal
circumstances, use 'nodetool drain' as part of stopping the node, before
stopping/killing the service/process.

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le mar. 18 juin 2019 à 23:43, ZAIDI, ASAD A <az192g@att.com> a écrit :

>
>
> I’m on environment with  apache Cassandra 3.11.1 with  java 1.8.0_144.
>
>
>
> One Node went OOM and crashed. Re-starting this crashed node is taking
> long time. Trace level debug log is showing messages like:
>
>
>
>
>
> Debug.log trace excerpt:
>
> ========================
>
>
>
> TRACE [main] 2019-06-18 21:30:43,449 LogTransaction.java:217 - Deleting
> /cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc-9337720-big-CompressionInfo.db
>
> TRACE [main] 2019-06-18 21:30:43,449 LogTransaction.java:217 - Deleting
> /cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc-9337720-big-Filter.db
>
> TRACE [main] 2019-06-18 21:30:43,449 LogTransaction.java:217 - Deleting
> /cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc-9337720-big-TOC.txt
>
> TRACE [main] 2019-06-18 21:30:43,455 LogTransaction.java:217 - Deleting
> /cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc_txn_compaction_642976c0-91c3-11e9-97bb-6b1dee397c3f.log
>
> TRACE [main] 2019-06-18 21:30:43,458 LogReplicaSet.java:67 - Added log
> file replica
> /cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc_txn_compaction_5a6c8c90-91cc-11e9-97bb-6b1dee397c3f.log
>
>
>
>
>
> Above messages are repeated for unique [mc-nnnn-* ] files. Such messages
> are repeating constantly.
>
>
>
> I’m seeking help here to find out what may be going on here , any hint to
> root cause and how I can quickly start the node. Thanks in advance.
>
>
>
> Regards/asad
>
>
>
>
>
>
>

Mime
View raw message