cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Lerer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10971) Compressed commit log has no backpressure and can OOM
Date Wed, 16 Mar 2016 17:07:33 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197690#comment-15197690
] 

Benjamin Lerer commented on CASSANDRA-10971:
--------------------------------------------

I ran the tests on CI for 3.5 and they were flapping.
I add a look at the tests and found 2 problems:
* Some tests were using {{new CommitLog()}} with the same directory that {{CommitLog.INSTANCE}}
which was causing 2 commit log instance to run at the same time with 2 differents configurations.
This was resulting on some commit log files not being deleted for the {{replay_StandardMmapped}}
test.
* As the unit tests are run in random orders the configuration changes made by the compression
and encrytion tests were affecting other test when {{resetUnsafe}} was used.

I fixed the problems by using only {{CommitLog.INSTANCE}} in all the tests and restoring the
initial configuration parameters after each test that was modifying them.

Ran the test on CI and it looks that we are good to go. \o/  
Thanks for all the work [~aweisberg]

> Compressed commit log has no backpressure and can OOM
> -----------------------------------------------------
>
>                 Key: CASSANDRA-10971
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10971
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local Write-Read Paths
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 3.0.x, 3.x
>
>
> I validated this via a unit test that slowed the ability of the log to drain to the filesystem.
The compressed commit log will keep allocating buffers pending compression until it OOMs.
> I have a fix that am not very happy with because the whole signal a thread to allocate
a segment that depends on a resource that may not be available results in some obtuse usage
of {{CompleatableFuture}} to rendezvous available buffers with {{CommitLogSegmentManager}}
thread waiting to finish constructing a new segment. The {{CLSM}} thread is in turn signaled
by the thread(s) that actually wants to write to the next segment, but aren't able to do it
themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message