cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeffrey Wang (JIRA)" <j...@apache.org>
Subject [jira] Updated: (CASSANDRA-2350) Races between schema changes and StorageService operations
Date Thu, 17 Mar 2011 18:52:29 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeffrey Wang updated CASSANDRA-2350:
------------------------------------

    Description: 
I only tested this on 0.7.0, but it judging by the 0.7.3 code (latest I've looked at) the
same thing should happen.

The case in particular that I ran into is this: I force a compaction for all CFs in a keyspace,
and while the compaction is happening I add another CF to the keyspace. I get the following
exception because the underlying set of CFs has changed while being iterated over.

{noformat}
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
        at java.util.HashMap$ValueIterator.next(Unknown Source)
        at java.util.Collections$UnmodifiableCollection$1.next(Unknown Source)
        at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1140)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source) 
{noformat}

The problem is a little more fundamental than that, though, as I believe any schema change
of CFs in the keyspace during one of these operations (e.g. flush, compaction, etc) has the
potential to cause a race. I'm not sure what would happen if the set of CFs to compact was
acquired and one of them was dropped before it had been compacted.

  was:
I only tested this on 0.7.0, but it judging by the 0.7.3 code (latest I've looked at) the
same thing should happen.

The case in particular that I ran into is this: I force a compaction for all CFs in a keyspace,
and while the compaction is happening I add another CF to the keyspace. I get the following
exception because the underlying set of CFs has changed while being iterated over.

{noformat}
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
        at java.util.HashMap$ValueIterator.next(Unknown Source)
        at java.util.Collections$UnmodifiableCollection$1.next(Unknown Source)
        at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1140)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source) 
{noformat}

The problem is a little for fundamental than that, though, as I believe any schema change
of CFs in the keyspace during one of these operations (e.g. flush, compaction, etc) have the
potential to cause a race. I haven't tried this out, but I'm not sure what would happen if
the set of CFs to compact was acquired and one of them was dropped before it had been compacted.


> Races between schema changes and StorageService operations
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-2350
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2350
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Jeffrey Wang
>
> I only tested this on 0.7.0, but it judging by the 0.7.3 code (latest I've looked at)
the same thing should happen.
> The case in particular that I ran into is this: I force a compaction for all CFs in a
keyspace, and while the compaction is happening I add another CF to the keyspace. I get the
following exception because the underlying set of CFs has changed while being iterated over.
> {noformat}
> java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
>         at java.util.HashMap$ValueIterator.next(Unknown Source)
>         at java.util.Collections$UnmodifiableCollection$1.next(Unknown Source)
>         at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1140)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>         at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
>         at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
>         at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
>         at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
>         at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
>         at sun.rmi.transport.Transport$1.run(Unknown Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at sun.rmi.transport.Transport.serviceCall(Unknown Source)
>         at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source) 
> {noformat}
> The problem is a little more fundamental than that, though, as I believe any schema change
of CFs in the keyspace during one of these operations (e.g. flush, compaction, etc) has the
potential to cause a race. I'm not sure what would happen if the set of CFs to compact was
acquired and one of them was dropped before it had been compacted.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message