cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13624) upgradesstables crashes with OOM when upgrading sstables with lots of range tombstones
Date Tue, 20 Jun 2017 21:18:01 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056479#comment-16056479
] 

Jeff Jirsa commented on CASSANDRA-13624:
----------------------------------------

Do you get the same OOM with the online version as you do with the offline?

> upgradesstables crashes with OOM when upgrading sstables with lots of range tombstones
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-13624
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13624
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Robin Mahony
>
> CASSANDRA-7953 can lead to range tombstones not properly being compacted. When trying
to upgrade from Cassandra 2.0.X to 2.1.X, running upgradesstables (either via nodetool OR
the offline version) does not work, as it crashes with OOM. This essentially means if you
are running a version of Cassandra 2.0.X for a period of time where lots of range tombstones
have been generated; you will be unable to upgrade to Cassandra 2.1.X following the normal
procedures. Hence upgrade from 2.0.X to 2.1.X is essentially broken if you hit CASSANDRA-7953.
> I hit this while trying to upgrade to Cassandra 2.1.17.
> Offline Version:
> # sstableupgrade storagegrid s3_usage_delta 
> Found 15 sstables that need upgrading. 
> Upgrading SSTableReader(path='/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-jb-20230-Data.db')

> ERROR 22:38:24,626 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6821300e)
to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@244363601:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79531
was not released before the reference was garbage collected 
> ERROR 22:38:24,631 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2891457f)
to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1230033823:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79532
was not released before the reference was garbage collected 
> Exception in thread "ScheduledTasks:1" Exception in thread "metrics-meter-tick-thread-2"
java.lang.OutOfMemoryError: Java heap space 
> at java.lang.Class.getName0(Native Method) 
> at java.lang.Class.getName(Class.java:642) 
> at java.lang.Throwable.toString(Throwable.java:479) 
> at java.lang.Throwable.<init>(Throwable.java:311) 
> at java.lang.Exception.<init>(Exception.java:102) 
> at java.util.concurrent.ExecutionException.<init>(ExecutionException.java:90) 
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) 
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) 
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:246)
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:210)
> at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:89)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
> at java.lang.Thread.run(Thread.java:745) 
> java.lang.OutOfMemoryError: Java heap space 
> java.lang.OutOfMemoryError: Java heap space 
> ERROR 22:41:21,080 Error in ThreadPoolExecutor 
> java.lang.OutOfMemoryError: Java heap space 
> ERROR 22:41:21,080 JVM state determined to be unstable. Exiting forcefully due to: 
> java.lang.OutOfMemoryError: Java heap space 
> root@DC1-SN-10-224-6-066:/var/local/cassandra/data/0 #



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message