cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-6879) ConcurrentModificationException while doing range slice query.
Date Fri, 21 Mar 2014 22:46:48 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-6879:
--------------------------------------

    Attachment: 6879.txt

I'm actually confused that this doesn't happen more often, because we invoke maybeResolveForRepair
every time we receive more responses than we need to satisfy consistencylevel -- and for range
slices it looks like we always send the request to all replicas (possibly restricted to local
DC).  So at CL.ONE I would expect to see this frequently.

Since we don't attempt read repair on range scans yet (CASSANDRA-967) it looks to me like
we can fix by simply restricting the replicas contacted to what is required for the consistencylevel.
 Then resolve() will only be called by the StorageProxy thread as intended, and there is no
race.


> ConcurrentModificationException while doing range slice query.
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-6879
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6879
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: 2.0.4
>            Reporter: Shao-Chuan Wang
>            Assignee: Mikhail Stepura
>             Fix For: 2.0.7
>
>         Attachments: 6879.txt
>
>
> The paging read request (either from thrift or native) would sporadically fail due to
a race condition between read repair and requesting thread waiting for read repair results
list. The READ_REPAIR is queued in ReadCallback.maybeResolveForRepair(), and it does not seem
to have guarantee that its resolve() method (which internally create RangeSliceResponseResolver.Reducer
and doing repairResults.addAll inside RangeSliceResponseResolver.Reducer) would be invoked
before the requesting thread starts waiting on resolver.repairResults. So, there is a small
window that the list is partially populated, while requesting thread starts waiting on repairResults.
I believe for the most of the time, the requesting thread is either wait for the entire repair
results or not waiting for repair results at all. The original intent here seems to be waiting
for repair results always (if the repair is triggered by repair chance).
> {code}
> ERROR [Native-Transport-Requests:70827] 2014-03-18 05:00:12,774 ErrorMessage.java (line
222) Unexpected exception during request
> java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
>         at java.util.ArrayList$Itr.next(ArrayList.java:831)
>         at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:423)
>         at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1583)
>         at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:188)
>         at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:163)
>         at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:58)
>         at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>         at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358)
>         at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131)
>         at org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>         at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>         at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> {code}
> {code}
> ERROR [Thrift:1] 2014-03-18 07:18:02,434 CustomTThreadPoolServer.java (line 212) Error
occurred during processing of message.
> java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
>         at java.util.ArrayList$Itr.next(ArrayList.java:831)
>         at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:423)
>         at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1583)
>         at org.apache.cassandra.service.pager.RangeSliceQueryPager.queryNextPage(RangeSliceQueryPager.java:85)
>         at org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:71)
>         at org.apache.cassandra.service.pager.RangeSliceQueryPager.fetchPage(RangeSliceQueryPager.java:36)
>         at org.apache.cassandra.cql3.statements.SelectStatement.pageCountQuery(SelectStatement.java:202)
>         at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:169)
>         at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:58)
>         at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>         at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>         at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:212)
>         at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
>         at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
>         at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message