cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Yeap <wkk1...@gmail.com>
Subject Re: Node Stuck while restarting
Date Sun, 29 May 2016 11:37:55 GMT
Hi Bhuvan, how big are your current commit logs in the failed node, and
what are the sizes MAX_HEAP_SIZE and HEAP_NEWSIZE?

Also the values of following properties in cassandra.yaml??

memtable_allocation_type
memtable_cleanup_threshold
memtable_flush_writers
memtable_heap_space_in_mb
memtable_offheap_space_in_mb


Regards,
Mike Yeap



On Sun, May 29, 2016 at 6:18 PM, Bhuvan Rawal <bhu1rawal@gmail.com> wrote:

> Hi,
>
> We are running a 6 Node cluster in 2 DC on DSC 3.0.3, with 3 Node each.
> One of the node was showing UNREACHABLE on other nodes in nodetool
> describecluster  and on that node it was showing all others UNREACHABLE and
> as a measure we restarted the node.
>
> But on doing that it is stuck possibly at with these messages in
> system.log:
>
> DEBUG [SlabPoolCleaner] 2016-05-29 14:07:28,156 ColumnFamilyStore.java:829
> - Enqueuing flush of batches: 226784704 (11%) on-heap, 0 (0%) off-heap
> DEBUG [main] 2016-05-29 14:07:28,576 CommitLogReplayer.java:415 -
> Replaying /commitlog/data/CommitLog-6-1464508993391.log (CL version 6,
> messaging version 10, compression null)
> DEBUG [main] 2016-05-29 14:07:28,781 ColumnFamilyStore.java:829 -
> Enqueuing flush of batches: 207333510 (10%) on-heap, 0 (0%) off-heap
>
> MemtablePostFlush / MemtableFlushWriter stages where it is stuck with
> pending messages.
> This has been the status of them as per *nodetool tpstats *for long.
> MemtablePostFlush                 Active - 1        pending - 52
> completed - 16
> MemtableFlushWriter               Active - 2        pending - 13
> completed - 15
>
>
> We restarted the node by setting log level to TRACE but in vain. What
> could be a possible contingency plan in such a scenario?
>
> Best Regards,
> Bhuvan
>
>

Mime
View raw message