cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Hust (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
Date Wed, 30 Sep 2015 20:18:04 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Hust updated CASSANDRA-10360:
------------------------------------
    Description: 
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck
in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
    at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
    at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
    at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
    at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
    at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
{code}

It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not surface the
issue.  It can be reproduced with the following (note -- changing jdks will need to be altered
if you aren't on OSX)
{code}
#!/bin/sh

echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)

ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10

for cver in 2.2 3.0
do
    echo "Draining all nodes"
    ccm node1 nodetool drain
    ccm stop

    echo "switching to java 8"
    export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

    echo "Upgrading to git:cassandra-$cver"
    ccm setdir -v git:cassandra-$cver
    ccm start
    echo "Sleeping to all version migrations"
    sleep 30
    echo "Upgrading sstables"
    ccm node1 nodetool upgradesstables
    ccm node1 nodetool upgradesstables system
    ccm node1 nodetool compact system

    ccm node1 stress write n=500K -rate threads=4 -mode native cql3
    sleep 10
done

echo "***********"
echo "instead of creating churn to cause compaction naturally forcing compaction of system
keyspace"
echo "***********"
ccm node1 nodetool compact system
ccm stop
{code}

The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables
system}}.  I'm puzzled by that since the script runs {{upgradesstables}} on each iteration.
 Is the system keyspace not effected by the command without arguments?

Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}

//CC [~enigmacurry]

  was:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck
in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
    at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
    at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
    at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
    at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
    at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
    at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
    at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
{code}

It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not surface the
issue.  It can be reproduced with the following (note -- changing jdks will need to be altered
if you aren't on OSX)
{code}
#!/bin/sh

echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)

ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10

for cver in 2.2 3.0
do
  echo "Draining all nodes"
  ccm node1 nodetool drain
  ccm stop

  echo "switching to java 8"
  export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

  echo "Upgrading to git:cassandra-$cver"
  ccm setdir -v git:cassandra-$cver
  ccm start
  echo "Sleeping to all version migrations"
  sleep 30
  echo "Upgrading sstables"
  ccm node1 nodetool upgradesstables

  ccm node1 stress write n=5K -rate threads=4 -mode native cql3
  sleep 10
done

echo "***********"
echo "instead of creating churn to cause compaction naturally forcing compaction of system
keyspace"
echo "***********"
ccm node1 nodetool compact system
ccm stop
{code}

The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables
system}}.  I'm puzzled by that since the script runs {{upgradesstables}} on each iteration.
 Is the system keyspace not effected by the command without arguments?

Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}

//CC [~enigmacurry]


> UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2
-> 3.0 upgrade
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10360
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Andrew Hust
>            Assignee: Sylvain Lebresne
>            Priority: Blocker
>             Fix For: 3.0.0 rc2
>
>         Attachments: size_estimates.tar.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
>     at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
>     at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
>     at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
>     at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
>     at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
>     at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>     at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
>     at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
>     at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
>     at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
>     at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
>     at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>     at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
>     at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>     at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not surface
the issue.  It can be reproduced with the following (note -- changing jdks will need to be
altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 2.2 3.0
> do
>     echo "Draining all nodes"
>     ccm node1 nodetool drain
>     ccm stop
>     echo "switching to java 8"
>     export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
>     echo "Upgrading to git:cassandra-$cver"
>     ccm setdir -v git:cassandra-$cver
>     ccm start
>     echo "Sleeping to all version migrations"
>     sleep 30
>     echo "Upgrading sstables"
>     ccm node1 nodetool upgradesstables
>     ccm node1 nodetool upgradesstables system
>     ccm node1 nodetool compact system
>     ccm node1 stress write n=500K -rate threads=4 -mode native cql3
>     sleep 10
> done
> echo "***********"
> echo "instead of creating churn to cause compaction naturally forcing compaction of system
keyspace"
> echo "***********"
> ccm node1 nodetool compact system
> ccm stop
> {code}
> The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables
system}}.  I'm puzzled by that since the script runs {{upgradesstables}} on each iteration.
 Is the system keyspace not effected by the command without arguments?
> Ran against:
> 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
> 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
> 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}
> //CC [~enigmacurry]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message