cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Will Hayworth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
Date Wed, 17 Feb 2016 07:36:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150049#comment-15150049
] 

Will Hayworth commented on CASSANDRA-11172:
-------------------------------------------

debug.log? I can go hunt through my compressed files but this quickly pushes the other logs
out of rotation. I've watched it happen. Thousands upon thousands of lines. And it's happened
for a bunch of different nodes. The solution has been to kill them and restart but it's temporary.

I just checked--all 20 zipped system.logs are filled with 40,000+ lines like the ones in my
sample file. I was going to upload some to you but they're just identical, including the earliest
lines I have:
{noformat}
INFO  [CompactionExecutor:35] 2016-02-17 01:59:03,111 LeveledManifest.java:438 - Adding high-level
(L0) BigTableReader(path='/var/lib/cassandra/data/segmentation/domain_events_by_event_domain_time-e81d74f0cd3a11e5aad8e7b84e29e52f/ma-3663-big-Data.db')
to candidates
INFO  [CompactionExecutor:37] 2016-02-17 01:59:03,112 LeveledManifest.java:438 - Adding high-level
(L3) BigTableReader(path='/var/lib/cassandra/data/segmentation/times_by_event_domain_user-e6112a30cd3a11e5ba896547d15a24f6/ma-4586-big-Data.db')
to candidates
INFO  [CompactionExecutor:35] 2016-02-17 01:59:03,276 LeveledManifest.java:438 - Adding high-level
(L0) BigTableReader(path='/var/lib/cassandra/data/segmentation/domain_events_by_event_domain_time-e81d74f0cd3a11e5aad8e7b84e29e52f/ma-3663-big-Data.db')
to candidates
INFO  [CompactionExecutor:37] 2016-02-17 01:59:03,276 LeveledManifest.java:438 - Adding high-level
(L3) BigTableReader(path='/var/lib/cassandra/data/segmentation/times_by_event_domain_user-e6112a30cd3a11e5ba896547d15a24f6/ma-4586-big-Data.db')
to candidates
{noformat}

> Infinite loop bug adding high-level SSTableReader in compaction
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-11172
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: DSE 4.x / Cassandra 2.1.11.969
>            Reporter: Jeff Ferland
>            Assignee: Marcus Eriksson
>         Attachments: beep.txt
>
>
> Observed that after a large repair on LCS that sometimes the system will enter an infinite
loop with vast amounts of logs lines recording, "Adding high-level (L${LEVEL}) SSTableReader(path='${TABLE}')
to candidates"
> This results in an outage of the node and eventual crashing. The log spam quickly rotates
out possibly useful earlier debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message