cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs
Date Fri, 29 Sep 2017 23:51:03 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186704#comment-16186704
] 

Jeff Jirsa commented on CASSANDRA-13123:
----------------------------------------

Hi folks,

Pretty sure this commit breaks {{CommitLogSegmentManagerTest}} - have seen a pretty sharp
rise in failures, and reverting this commit seems to solve them.

{code}
    [junit] INFO  23:34:52 Initializing CommitLogTest.Standard1
    [junit] INFO  23:34:52 Initializing CommitLogTest.Standard2
    [junit] ------------- ---------------- ---------------
    [junit] Testcase: testShutdownWithPendingTasks(org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest):
FAILED
    [junit] null
    [junit] junit.framework.AssertionFailedError
    [junit] 	at org.apache.cassandra.db.Keyspace.open(Keyspace.java:105)
    [junit] 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest.testShutdownWithPendingTasks(CommitLogSegmentManagerTest.java:147)
    [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$10.evaluate(BMUnitRunner.java:371)
    [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:241)
    [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:75)
    [junit]
    [junit]
    [junit] Testcase: testCompressedCommitLogBackpressure(org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest):
FAILED
    [junit] expected:<3> but was:<1>
    [junit] junit.framework.AssertionFailedError: expected:<3> but was:<1>
    [junit] 	at org.apache.cassandra.Util.spinAssertEquals(Util.java:535)
    [junit] 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest.testCompressedCommitLogBackpressure(CommitLogSegmentManagerTest.java:112)
    [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$9.evaluate(BMUnitRunner.java:342)
    [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:241)
    [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:75)
    [junit]
    [junit]
    [junit] Test org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest FAILED
   [delete] Deleting directory /Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/commitlog:0
   [delete] Deleting directory /Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/data:0
   [delete] Deleting directory /Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/saved_caches:0
[junitreport] Processing /Users/jjirsa/Desktop/Dev/cassandra/build/test/TESTS-TestSuites.xml
to /var/folders/nq/4w83hn7s3h13dc5wxmvcdn9w0000gn/T/null1048031913
[junitreport] Loading stylesheet jar:file:/usr/local/ant/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 256ms
[junitreport] Deleting: /var/folders/nq/4w83hn7s3h13dc5wxmvcdn9w0000gn/T/null1048031913
{code}


> Draining a node might fail to delete all inactive commitlogs
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-13123
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local Write-Read Paths
>            Reporter: Jan Urbański
>            Assignee: Jan Urbański
>             Fix For: 3.0.15, 3.11.1, 4.0
>
>         Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive commitlogs
are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down the CommitLogSegmentManager.
This has the effect of discarding any pending management tasks it might have, like the removal
of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind after a drain
and a lengthy recovery after a restart. With a fleet of dozens of nodes, each of them leaving
several GB of commitlogs after a drain and taking up to two minutes to recover them on restart,
the additional time required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message