cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
Date Tue, 29 Aug 2017 18:05:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145799#comment-16145799
] 

Ariel Weisberg commented on CASSANDRA-13785:
--------------------------------------------

New stuff and summarizing.
* So you are the unlucky person to catch SafeMemoryWriter misbehaving. If we want our test
hygiene to be good you would be the person to write a basic set of unit tests for it. Go through
each external method in SafeMemoryWriter and test it's API (null handling, largest value,
smallest value, negative value, 0) and internal state (0 length, empty buffer, full buffer,
etc.).
* This will still hit the same issue if the decorated key size is larger than expected right?
I know almost everyone uses fixed length keys, but it looks fragile to not at least fail fast
if they use variable length keys or if the size changes in future iterations.
* This is a bug in {{SafeMemoryWriter.length()}}?
* Even if this throws AssertionError it still shouldn't leak memory right? We should log the
error, but not leak. I think (I'll check) that we don't treat assertion failures as fatal
and crash the process.


> Compaction fails for SSTables with large number of keys
> -------------------------------------------------------
>
>                 Key: CASSANDRA-13785
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Jay Zhuang
>            Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
Memory@[0..d8123468)] was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
Memory@[0..d8123468)] was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
Memory@[0..d8123468)] was not released before the reference was garbage collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the compacted SSTable
has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will fail to create the IndexSummary:
[IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
with {{> Integer.MAX_VALUE}} space, it returns the tailing {{Integer.MAX_VALUE}} space
[SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
which makes the first [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
not 0. So the assert fails here: [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message