cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Mitchell <mitch...@es.net>
Subject Re: Weird timeout errors
Date Fri, 13 Jun 2014 09:58:02 GMT

On Jun 12, 2014, at 11:39 AM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Thu, Jun 12, 2014 at 10:29 AM, David Mitchell <mitchell@es.net> wrote:
>         session.execute("""insert into raw_data (key,column1,value) values (%s,%s,%s)""",
> ...
> and then delete them like so:
>     session.execute("""delete from raw_data where key = %s""",(path,))
> ...
> and then try to select from that key like so:
>     rows = session.execute("""select * from raw_data where key = %s limit 5""",(path,))
> ...
> This works fine for small values of count, say 1000 or so. If I do this with 500,000
rows I end up in a state where any attempt to query the key results in a timeout error from
Cassandra. The only way I seem to be able to clear it up is to completely stop and start the
Cassandra server. I’m running version 2.0.7 and I don’t think this was happening under
2.0.1. Is this a known bug? Any ideas what might be causing this? Thanks in advance,
> 
> Set Tracing on, it should be informative.
> 
> All operations in Cassandra are subject to timeouts which default to a low number of
seconds. Deleting data results in tombstones. Requesting the (probably) 2Mn objects which
are created when you create and then delete 500,000 rows is relatively unlikely to complete
in this amount of time.
> 
> Because of these timeouts, in general anything that requests large number of partitions
should be handled with the new async stuff in the drivers, not with gigantic batch requests.

It does look like the excessive number of tombstones is the culprit. Thanks for pointing me
towards that. The trace included:
 Scanned over 100000 tombstones; query aborted (see tombstone_fail_threshold) | 02:18:53,436
| 10.10.252.14 |          44050

The apparent behavior of the nodes to simply not answer and let a timeout come back to the
user is a little frustrating but I can adapt. The next question I have is how to clean out
the tombstones? I would expect ‘nodetool compact’ to do this perhaps? My experience has
been that restarting cassandra clears up the issue (until the next big delete of course).
I would expect there to be some way to clean them out with the server up. I don’t do big
row deletions that often but after I do I need to be able to continue working on those row
keys without errors. Thanks in advance,

-David Mitchell


> 
> =Rob
> 

--
David Mitchell, Network Engineer
Energy Sciences Network (ESnet)
Lawrence Berkeley National Laboratory (LBL)
Email:mitchell@es.net  Phone:(510) 936-0720



Mime
View raw message