incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Hinted Handoff runs every ten minutes
Date Sun, 21 Oct 2012 23:44:08 GMT
I *think* this may be ghost rows which have not being compacted.

How many SSTables are on disk for the HintedHandoff CF ?

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 19/10/2012, at 7:16 AM, David Daeschler <david.daeschler@gmail.com> wrote:

> Hi Steve,
> 
> Also confirming this. After having a node go down on Cassandra 1.0.8
> there seems to be hinted handoff between two of our 4 nodes every 10
> minutes. Our setup also shows 0 rows. It does not appear to have any
> effect on the operation of the ring, just fills up the log files.
> 
> - David
> 
> 
> 
> On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce <spierce@verifyle.com> wrote:
>> I installed Cassandra on three nodes. I then ran a test suite against them
>> to generate load. The test suite is designed to generate the same type of
>> load that we plan to have in production. As one of many tests, I reset one
>> of the nodes to check the failure/recovery modes.  Cassandra worked just
>> fine.
>> 
>> 
>> 
>> I stopped the load generation, and got distracted with some other
>> project/problem. A few days later, I noticed something strange on one of the
>> nodes. On this node hinted handoff starts every ten minutes, and while it
>> seems to finish without any errors, it will be started again in ten minutes.
>> None of the nodes has any traffic, and hasn’t for several days. I checked
>> the logs, and this goes back to the initial failure/recovery testing:
>> 
>> 
>> 
>> INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
>> (line 294) Started hinted handoff for token:
>> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>> 
>> INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
>> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>> 
>> INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
>> (line 294) Started hinted handoff for token:
>> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>> 
>> INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
>> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>> 
>> INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
>> (line 294) Started hinted handoff for token:
>> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>> 
>> INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
>> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>> 
>> 
>> 
>> The other nodes are happy and don’t show this behavior. All the test data is
>> readable, and everything is fine, but I’m curious why hinted handoff is
>> running on one node all the time.
>> 
>> 
>> 
>> I searched the bug database, and I found a bug that seems to have the same
>> symptoms:
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-3733
>> 
>> Although it’s been marked fixed in 0.6, this describes my problem exactly.
>> 
>> 
>> 
>> I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:
>> 
>> http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
>> 
>> 
>> 
>> Is anyone else seeing this behavior? What can I do to provide more
>> information?
>> 
>> 
>> 
>> Steve
>> 
>> 
> 
> 
> 
> -- 
> David Daeschler


Mime
View raw message