cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Zhuang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13785) Compaction fails for SSTables with large number of keys
Date Fri, 25 Aug 2017 00:24:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140977#comment-16140977
] 

Jay Zhuang edited comment on CASSANDRA-13785 at 8/25/17 12:23 AM:
------------------------------------------------------------------

[~rha] that makes sense. I updated the patch to define the magic number.
Basically, the number is coming from here: [IndexSummaryBuilder.java:112|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
We potentially could support more entries if the partition key size is smaller than {{32}},
but I think it's still good to increase the {{[effectiveMinInterval|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L96]}}
to reduce the summary size automatically in that case.
Also the "large number of keys" more likely to have larger partition key size (like {{32}}
or even larger).

| branch | dTest |
| [13785-3.0|https://github.com/cooldoger/cassandra/tree/13785-3.0] | [!https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.0.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.0]
|
| [13785-3.11|https://github.com/cooldoger/cassandra/tree/13785-3.11] | [!https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.11]
|
| [13785-trunk|https://github.com/cooldoger/cassandra/tree/13785-trunk] | [!https://circleci.com/gh/cooldoger/cassandra/tree/13785-trunk.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-trunk]
|



was (Author: jay.zhuang):
[~rha] that makes sense. I updated the patch to define the magic number.
Basically, the number is coming from here: [IndexSummaryBuilder.java:112|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
We potentially could support more entries if the partition key size is smaller than {{32}},
but I think it's still good to increase the {{[effectiveMinInterval|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L96]}}
to reduce the summary size automatically in that case.
Also the "large number of keys" more likely have larger partition key size (like {{32}} or
even larger).

| branch | dTest |
| [13785-3.0|https://github.com/cooldoger/cassandra/tree/13785-3.0] | [!https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.0.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.0]
|
| [13785-3.11|https://github.com/cooldoger/cassandra/tree/13785-3.11] | [!https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-3.11]
|
| [13785-trunk|https://github.com/cooldoger/cassandra/tree/13785-trunk] | [!https://circleci.com/gh/cooldoger/cassandra/tree/13785-trunk.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13785-trunk]
|


> Compaction fails for SSTables with large number of keys
> -------------------------------------------------------
>
>                 Key: CASSANDRA-13785
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13785
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Jay Zhuang
>            Assignee: Jay Zhuang
>
> Every a few minutes there're "LEAK DTECTED" messages in the log:
> {noformat}
> ERROR [Reference-Reaper:1] 2017-08-18 17:18:40,357 Ref.java:223 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3ed22d7) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1022568824:[Memory@[0..159b6ba4),
Memory@[0..d8123468)] was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:20:49,693 Ref.java:223 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6470405b) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@97898152:[Memory@[0..159b6ba4),
Memory@[0..d8123468)] was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2017-08-18 17:22:38,519 Ref.java:223 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6fc4af5f) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1247404854:[Memory@[0..159b6ba4),
Memory@[0..d8123468)] was not released before the reference was garbage collected
> {noformat}
> Debugged the issue and found it's triggered by failed compactions, if the compacted SSTable
has more than 51m {{Integer.MAX_VALUE / 40}}) keys, it will fail to create the IndexSummary:
[IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84].
> Cassandra compaction tried to compact every a few minutes and keeps failing.
> The root cause is while [creating SafeMemoryWriter|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L112]
with {{> Integer.MAX_VALUE}} space, it returns the tailing {{Integer.MAX_VALUE}} space
[SafeMemoryWriter.java:83|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/util/SafeMemoryWriter.java#L83],
which makes the first [entries.length()|https://github.com/apache/cassandra/blob/6a1b1f26b7174e8c9bf86a96514ab626ce2a4117/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java#L173]
not 0. So the assert fails here: [IndexSummary:84|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/IndexSummary.java#L84]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message