cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Kinder (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
Date Mon, 23 Oct 2017 22:44:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215984#comment-16215984
] 

Dan Kinder commented on CASSANDRA-13973:
----------------------------------------

[~jjirsa]

This happens on about 12 out of 16 nodes, much greater than {{RF}}. (No real need to anonymize
the names.)

{noformat}
$ nodetool tablehistograms walker links
walker/links histograms
Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)               
  
50%             4.00             51.01        1155149.91               258               
 6
75%             5.00             73.46        3449259.15              1597               
42
95%             5.00            152.32        7152383.77             73457              2299
98%             5.00            182.79        8582860.53            315852              9887
99%             5.00            219.34       53142810.15            943127             24601
Min             0.00              8.24             88.15                43               
 0
Max             5.00        1155149.91       53142810.15      394855527800         962624926

$ nodetool tablestats walker.links
Keyspace: walker
        Read Count: 72932
        Read Latency: 834.917616670323 ms.
        Write Count: 15501219
        Write Latency: 0.4392410533003888 ms.
        Pending Flushes: 0
                Table: links
                SSTable count: 14269
                SSTables in each level: [0, 9, 68, 349, 2843, 11000, 0, 0, 0]
                Space used (live): 5310634842695
                Space used (total): 5310634842695
                Space used by snapshots (total): 12868175419329
                Off heap memory used (total): 16339462
                SSTable Compression Ratio: 0.0
                Number of partitions (estimate): 20112302
                Memtable cell count: 2216835
                Memtable data size: 347672840
                Memtable off heap memory used: 0
                Memtable switch count: 4
                Local read count: 80395
                Local read latency: 747.603 ms
                Local write count: 16891401
                Local write latency: 0.118 ms
                Pending flushes: 0
                Bloom filter false positives: 178
                Bloom filter false ratio: 0.00081
                Bloom filter space used: 14912616
                Bloom filter off heap memory used: 14798464
                Index summary off heap memory used: 1540998
                Compression metadata off heap memory used: 0
                Compacted partition minimum bytes: 43
                Compacted partition maximum bytes: 394855527800
                Compacted partition mean bytes: 249068
                Average live cells per slice (last five minutes): 11010.451219512195
                Maximum live cells per slice (last five minutes): 11864
                Average tombstones per slice (last five minutes): 1.0
                Maximum tombstones per slice (last five minutes): 1
{noformat}



> IllegalArgumentException in upgradesstables compaction
> ------------------------------------------------------
>
>                 Key: CASSANDRA-13973
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try to run upgradesstables,
most of them upgrade fine but I see the exception below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I assumed it is
not identical. The various nodes this happens on all give the same trace.
> Might be notable that this is an analytics cluster with some large partitions, in the
GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message