Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F21B4186DA for ; Wed, 30 Sep 2015 20:18:10 +0000 (UTC) Received: (qmail 7256 invoked by uid 500); 30 Sep 2015 20:18:04 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 7218 invoked by uid 500); 30 Sep 2015 20:18:04 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 7204 invoked by uid 99); 30 Sep 2015 20:18:04 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2015 20:18:04 +0000 Date: Wed, 30 Sep 2015 20:18:04 +0000 (UTC) From: "Andrew Hust (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Hust updated CASSANDRA-10360: ------------------------------------ Description: When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: {code} java.lang.UnsupportedOperationException at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) {code} #!/bin/sh echo "using java7 for cassandra-2.1 instance" export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) ccm stop ccm remove upgrade_nodes ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes ccm start ccm node1 stress write n=500K -rate threads=4 -mode native cql3 sleep 10 for cver in 2.2 3.0 do echo "Draining all nodes" ccm node1 nodetool drain ccm stop echo "switching to java 8" export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) echo "Upgrading to git:cassandra-$cver" ccm setdir -v git:cassandra-$cver ccm start echo "Sleeping to all version migrations" sleep 30 echo "Upgrading sstables" ccm node1 nodetool upgradesstables ccm node1 nodetool upgradesstables system ccm node1 nodetool compact system ccm node1 stress write n=500K -rate threads=4 -mode native cql3 sleep 10 done echo "***********" echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" echo "***********" ccm node1 nodetool compact system ccm stop {code} The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? Ran against: 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} //CC [~enigmacurry] was: When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: {code} java.lang.UnsupportedOperationException at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) {code} #!/bin/sh echo "using java7 for cassandra-2.1 instance" export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) ccm stop ccm remove upgrade_nodes ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes ccm start ccm node1 stress write n=5K -rate threads=4 -mode native cql3 sleep 10 for cver in 2.2 3.0 do echo "Draining all nodes" ccm node1 nodetool drain ccm stop echo "switching to java 8" export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) echo "Upgrading to git:cassandra-$cver" ccm setdir -v git:cassandra-$cver ccm start echo "Sleeping to all version migrations" sleep 30 echo "Upgrading sstables" ccm node1 nodetool upgradesstables ccm node1 stress write n=5K -rate threads=4 -mode native cql3 sleep 10 done echo "***********" echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" echo "***********" ccm node1 nodetool compact system ccm stop {code} The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? Ran against: 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} //CC [~enigmacurry] > UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade > --------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug > Reporter: Andrew Hust > Assignee: Sylvain Lebresne > Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.tar.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > for cver in 2.2 3.0 > do > echo "Draining all nodes" > ccm node1 nodetool drain > ccm stop > echo "switching to java 8" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) > echo "Upgrading to git:cassandra-$cver" > ccm setdir -v git:cassandra-$cver > ccm start > echo "Sleeping to all version migrations" > sleep 30 > echo "Upgrading sstables" > ccm node1 nodetool upgradesstables > ccm node1 nodetool upgradesstables system > ccm node1 nodetool compact system > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > done > echo "***********" > echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" > echo "***********" > ccm node1 nodetool compact system > ccm stop > {code} > The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? > Ran against: > 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} > 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} > 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} > //CC [~enigmacurry] -- This message was sent by Atlassian JIRA (v6.3.4#6332)