cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Miranda <dan...@cobli.co>
Subject Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)
Date Wed, 22 Mar 2017 20:34:28 GMT
I found out the problem is conditioned to having the row cache enabled.
Whenever a query would return an empty result set in a particular table, it
would fail instead with the exception being thrown in all all nodes.

Disabling the row cache for that particular table fixes the problem.
Re-enabling has not caused the issue again yet. I do not have row cache
saving enabled, and the issue persisted between node restarts, which it's
somewhat strange.

I couldn't reproduce the issue with any other tables. I can produce a
sanitized dump of the SSTable if a developer desires.

--
Regards,
Daniel

On Wed, Mar 22, 2017 at 2:33 PM Daniel Miranda <daniel@cobli.co> wrote:

> Thank you for the pointer Michael, I'll try to investigate if this is the
> same bug I am seeing.
>
> I am afraid it might not be, since I'm observing the error periodically,
> not just during compactions, and the traceback seems different.
>
> Regards,
> Daniel
>
> On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler <michael@pbandjelly.org>
> wrote:
>
> Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which
> shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed
> on there, which you might investigate, as well.
>
> --
> Kind regards,
> Michael
>
> On 03/22/2017 11:21 AM, Daniel Miranda wrote:
> > Greetings,
> >
> > Recently I've started to see the an assertion (traceback follows at the
> > end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster
> > (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all
> > nodes. Repairs run fine without indicating any errors.
> >
> > I can't seem to find any information about it from someone else or any
> > bug reports.
> >
> > Should I bother running an SSTable scrub? Is it a known issue that is
> > fixed in subsequent versions?
> >
> > Thanks in advance,
> > Daniel
> >
> > ---
> > WARN  [SharedPool-Worker-1] 2017-03-16 18:54:35,587
> > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
> > thread Thread[SharedPool-Worker-1,5,main]: {}
> > java.lang.AssertionError: null
> >         at
> >
> org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(UnfilteredRowIterators.java:157)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> >
> org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:420)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> >
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:324)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> >
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> >
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > ~[na:1.8.0_111]
> >         at
> >
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> > ~[apache-cassandra-3.5.jar:3.5]
> >         at
> >
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> > [apache-cassandra-3.5.jar:3.5]
> >         at
> > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> > [apache-cassandra-3.5.jar:3.5]
> >         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> > --
> > *Daniel Miranda*
> >
> > DevOps Engineering Intern
> > (11) 991959845
> > www.cobli.co <http://www.cobli.co/>
>
>

Mime
View raw message