incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kais Ahmed <k...@neteck-fr.com>
Subject Re: Crash with TombstoneOverwhelmingException
Date Fri, 27 Dec 2013 15:58:05 GMT
You can read the comments about this new feature here :

https://issues.apache.org/jira/browse/CASSANDRA-6117


2013/12/27 Kais Ahmed <kais@neteck-fr.com>

> This threshold is to prevent bad performance, you can increase the value
>
>
> 2013/12/27 Sanjeeth Kumar <sanjeeth@exotel.in>
>
>> Thanks for the replies.
>> I dont think this is just a warning , incorrectly logged as an error.
>> Everytime there is a crash, this is the exact traceback I see in the logs.
>> I just browsed through the code and the code throws
>> a TombstoneOverwhelmingException exception in these situations and I did
>> not see this being caught and handled some place. I might be wrong though.
>>
>> But I would also like to understand why this threshold value is important
>> , so that I can set a right threshold.
>>
>> - Sanjeeth
>>
>>
>> On Fri, Dec 27, 2013 at 11:33 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>>
>>> I do not think the feature is supposed to crash the server. It could be
>>> that the message is the logs and the crash is not related to this message.
>>> WARN might be a better logging level for any message, even though the first
>>> threshold is WARN and the second is FAIL. ERROR is usually something more
>>> dramatic.
>>>
>>>
>>> On Wed, Dec 25, 2013 at 1:02 PM, Laing, Michael <
>>> michael.laing@nytimes.com> wrote:
>>>
>>>> It's a feature:
>>>>
>>>> In the stock cassandra.yaml file for 2.03 see:
>>>>
>>>>  # When executing a scan, within or across a partition, we need to keep
>>>>> the
>>>>> # tombstones seen in memory so we can return them to the coordinator,
>>>>> which
>>>>> # will use them to make sure other replicas also know about the
>>>>> deleted rows.
>>>>> # With workloads that generate a lot of tombstones, this can cause
>>>>> performance
>>>>> # problems and even exaust the server heap.
>>>>> # (
>>>>> http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
>>>>> )
>>>>> # Adjust the thresholds here if you understand the dangers and want to
>>>>> # scan more tombstones anyway.  These thresholds may also be adjusted
>>>>> at runtime
>>>>> # using the StorageService mbean.
>>>>> tombstone_warn_threshold: 1000
>>>>> tombstone_failure_threshold: 100000
>>>>
>>>>
>>>> You are hitting the failure threshold.
>>>>
>>>> ml
>>>>
>>>>
>>>> On Wed, Dec 25, 2013 at 12:17 PM, Rahul Menon <rahul@apigee.com> wrote:
>>>>
>>>>> Sanjeeth,
>>>>>
>>>>> Looks like the error is being populated from the hintedhandoff, what
>>>>> is the size of your hints cf?
>>>>>
>>>>> Thanks
>>>>> Rahul
>>>>>
>>>>>
>>>>> On Wed, Dec 25, 2013 at 8:54 PM, Sanjeeth Kumar <sanjeeth@exotel.in>wrote:
>>>>>
>>>>>> Hi all,
>>>>>>   One of my cassandra nodes crashes with the following exception
>>>>>> periodically -
>>>>>> ERROR [HintedHandoff:33] 2013-12-25 20:29:22,276
>>>>>> SliceQueryFilter.java (line 200) Scanned over 100000 tombstones;
query
>>>>>> aborted (see tombstone_fail_thr
>>>>>> eshold)
>>>>>> ERROR [HintedHandoff:33] 2013-12-25 20:29:22,278 CassandraDaemon.java
>>>>>> (line 187) Exception in thread Thread[HintedHandoff:33,1,main]
>>>>>> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
>>>>>>         at
>>>>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
>>>>>>         at
>>>>>> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
>>>>>>         at
>>>>>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
>>>>>>         at
>>>>>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
>>>>>>         at
>>>>>> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
>>>>>>         at
>>>>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>         at java.lang.Thread.run(Thread.java:744)
>>>>>>
>>>>>> Why does this happen? Does this relate to any incorrect config value?
>>>>>>
>>>>>> The Cassandra Version I'm running is
>>>>>> ReleaseVersion: 2.0.3
>>>>>>
>>>>>> - Sanjeeth
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message