Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CADA318706 for ; Wed, 30 Sep 2015 20:22:04 +0000 (UTC) Received: (qmail 25363 invoked by uid 500); 30 Sep 2015 20:22:04 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 25329 invoked by uid 500); 30 Sep 2015 20:22:04 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 25315 invoked by uid 99); 30 Sep 2015 20:22:04 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2015 20:22:04 +0000 Date: Wed, 30 Sep 2015 20:22:04 +0000 (UTC) From: "Andrew Hust (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937461#comment-14937461 ] Andrew Hust edited comment on CASSANDRA-10360 at 9/30/15 8:21 PM: ------------------------------------------------------------------ I have now reproduced this issue when upgrading from 2.1 to 3.0 when doing a rolling upgrade with cstar_perf. Attached site_estimates sstables and system log from one of the nodes. I'll work on getting a isolated script to reproduce. UPDATE: The above shell script has been updated to show failure from 2.1 -> 3.0. It just needed more activity on the stress operation to trigger the issue. Change line {{for cver in 3.0}} to {{for cver in 2.2 3.0}} for the original upgrade path. was (Author: nutbunnies): I have now reproduced this issue when upgrading from 2.1 to 3.0 when doing a rolling upgrade with cstar_perf. Attached site_estimates sstables and system log from one of the nodes. I'll work on getting a isolated script to reproduce. > UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade > --------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug > Reporter: Andrew Hust > Assignee: Sylvain Lebresne > Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.tar.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > for cver in 2.2 3.0 > do > echo "Draining all nodes" > ccm node1 nodetool drain > ccm stop > echo "switching to java 8" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) > echo "Upgrading to git:cassandra-$cver" > ccm setdir -v git:cassandra-$cver > ccm start > echo "Sleeping to all version migrations" > sleep 30 > echo "Upgrading sstables" > ccm node1 nodetool upgradesstables > ccm node1 nodetool upgradesstables system > ccm node1 nodetool compact system > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > done > echo "***********" > echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" > echo "***********" > ccm node1 nodetool compact system > ccm stop > {code} > The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? > Ran against: > 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} > 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} > 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} > //CC [~enigmacurry] -- This message was sent by Atlassian JIRA (v6.3.4#6332)