incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Crash with TombstoneOverwhelmingException
Date Wed, 25 Dec 2013 18:01:58 GMT
I have to hijack this thread. There seem to be many problems with the 2.0.3
release.  If this exception is being generated by hinted-handoff, I could
understand where it is coming from. If you have many hints and many
tombstones then this new feature interacts with the hint delivery process,
in a bad way.

If I understand the feature correctly this feature should always be off for
the hints, because the regardless of how many tombstones are in the hints
this rule should not apply.

I want to bring up these questions:

Outside of passing all unit tests, factors into the release voting process?
What other type of extended real world testing should be done to find bugs
like this one that unit testing wont?

Not trying to call anyone out this feature/bug. I totally understand why
you would want a warning, or want to opt out of a read scanning over a
massive number of tombstones, and I think it is a smart feature. But what I
want more is to trust that every release is battle tested.

Here is a whack y idea that I am half serious about. Make a CMS for
http://cassndra.apache.org  that back ends it's data and reporting into
cassandra. No release unless Cassanda db that servers the site is upgraded
first. :)


On Wed, Dec 25, 2013 at 12:17 PM, Rahul Menon <rahul@apigee.com> wrote:

> Sanjeeth,
>
> Looks like the error is being populated from the hintedhandoff, what is
> the size of your hints cf?
>
> Thanks
> Rahul
>
>
> On Wed, Dec 25, 2013 at 8:54 PM, Sanjeeth Kumar <sanjeeth@exotel.in>wrote:
>
>> Hi all,
>>   One of my cassandra nodes crashes with the following exception
>> periodically -
>> ERROR [HintedHandoff:33] 2013-12-25 20:29:22,276 SliceQueryFilter.java
>> (line 200) Scanned over 100000 tombstones; query aborted (see
>> tombstone_fail_thr
>> eshold)
>> ERROR [HintedHandoff:33] 2013-12-25 20:29:22,278 CassandraDaemon.java
>> (line 187) Exception in thread Thread[HintedHandoff:33,1,main]
>> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
>>         at
>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
>>         at
>> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
>>         at
>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
>>         at
>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
>>         at
>> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
>>         at
>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
>>         at
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
>>         at
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
>>         at
>> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
>>         at
>> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
>>         at
>> org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
>>         at
>> org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:744)
>>
>> Why does this happen? Does this relate to any incorrect config value?
>>
>> The Cassandra Version I'm running is
>> ReleaseVersion: 2.0.3
>>
>> - Sanjeeth
>>
>>
>

Mime
View raw message