cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Hust (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
Date Wed, 16 Sep 2015 22:10:45 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Hust updated CASSANDRA-10360:
------------------------------------
    Description: 
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck
in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
    at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
    at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
    at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
    at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
    at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
{code}

It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not surface the
issue.  It can be reproduced with the following (note -- changing jdks will need to be altered
if you aren't on OSX)
{code}
#!/bin/sh

echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)

ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10

for cver in 2.2 3.0
do
  echo "Draining all nodes"
  ccm node1 nodetool drain
  ccm stop

  echo "switching to java 8"
  export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

  echo "Upgrading to git:cassandra-$cver"
  ccm setdir -v git:cassandra-$cver
  ccm start
  echo "Sleeping to all version migrations"
  sleep 30
  echo "Upgrading sstables"
  ccm node1 nodetool upgradesstables

  ccm node1 stress write n=5K -rate threads=4 -mode native cql3
  sleep 10
done

echo "***********"
echo "instead of creating churn to cause compaction naturally forcing compaction of system
keyspace"
echo "***********"
ccm node1 nodetool compact system
ccm stop
{code}

The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables
system}}.  I'm puzzled by that since the script runs {{upgradesstables}} on each iteration.
 Is the system keyspace not effected by the command without arguments?

Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}

//CC [~enigmacurry]

  was:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck
in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
    at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
    at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
    at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
    at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
    at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
{code}

It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not surface the
issue.  It can be reproduced with the following (note -- changing jdks will need to be altered
if you aren't on OSX)
{code}
#!/bin/sh

echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)

ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10

for cver in 2.2 3.0
do
  echo "Draining all nodes"
  ccm node1 nodetool drain
  ccm stop

  echo "switching to java 8"
  export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

  echo "Upgrading to git:cassandra-$cver"
  ccm setdir -v git:cassandra-$cver
  ccm start
  echo "Sleeping to all version migrations"
  sleep 30
  echo "Upgrading sstables"
  ccm node1 nodetool upgradesstables

  ccm node1 stress write n=5K -rate threads=4 -mode native cql3
  sleep 10
done

echo "***********"
echo "instead of creating churn to cause compaction naturally forcing compaction of system
keyspace"
echo "***********"
ccm node1 nodetool compact system
ccm stop
{code}

The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables
system}}.  I'm puzzled by that since the script runs {{upgradesstables}} on each iteration.
 Is the system keyspace not effected by the command without arguments?

Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}


> UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2
-> 3.0 upgrade
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10360
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Andrew Hust
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
>     at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
>     at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
>     at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
>     at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
>     at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
>     at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
>     at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
>     at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
>     at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
>     at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
>     at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>     at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
>     at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>     at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not surface
the issue.  It can be reproduced with the following (note -- changing jdks will need to be
altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=5K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 2.2 3.0
> do
>   echo "Draining all nodes"
>   ccm node1 nodetool drain
>   ccm stop
>   echo "switching to java 8"
>   export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
>   echo "Upgrading to git:cassandra-$cver"
>   ccm setdir -v git:cassandra-$cver
>   ccm start
>   echo "Sleeping to all version migrations"
>   sleep 30
>   echo "Upgrading sstables"
>   ccm node1 nodetool upgradesstables
>   ccm node1 stress write n=5K -rate threads=4 -mode native cql3
>   sleep 10
> done
> echo "***********"
> echo "instead of creating churn to cause compaction naturally forcing compaction of system
keyspace"
> echo "***********"
> ccm node1 nodetool compact system
> ccm stop
> {code}
> The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables
system}}.  I'm puzzled by that since the script runs {{upgradesstables}} on each iteration.
 Is the system keyspace not effected by the command without arguments?
> Ran against:
> 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
> 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
> 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}
> //CC [~enigmacurry]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message