cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Fraenkel <ta...@tok-media.com>
Subject Re: Hinted Handoff runs every ten minutes
Date Mon, 22 Oct 2012 13:05:23 GMT
Hi!
I am having the same issue on 1.0.8.
Checked number of SSTables, on two nodes I have 1 (on each) and on 1 node I
have none.
Thanks,

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Mon, Oct 22, 2012 at 1:44 AM, aaron morton <aaron@thelastpickle.com>wrote:

> I *think* this may be ghost rows which have not being compacted.
>
> How many SSTables are on disk for the HintedHandoff CF ?
>
> Cheers
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/10/2012, at 7:16 AM, David Daeschler <david.daeschler@gmail.com>
> wrote:
>
> Hi Steve,
>
> Also confirming this. After having a node go down on Cassandra 1.0.8
> there seems to be hinted handoff between two of our 4 nodes every 10
> minutes. Our setup also shows 0 rows. It does not appear to have any
> effect on the operation of the ring, just fills up the log files.
>
> - David
>
>
>
> On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce <spierce@verifyle.com>
> wrote:
>
> I installed Cassandra on three nodes. I then ran a test suite against them
> to generate load. The test suite is designed to generate the same type of
> load that we plan to have in production. As one of many tests, I reset one
> of the nodes to check the failure/recovery modes.  Cassandra worked just
> fine.
>
>
>
> I stopped the load generation, and got distracted with some other
> project/problem. A few days later, I noticed something strange on one of
> the
> nodes. On this node hinted handoff starts every ten minutes, and while it
> seems to finish without any errors, it will be started again in ten
> minutes.
> None of the nodes has any traffic, and hasn’t for several days. I checked
> the logs, and this goes back to the initial failure/recovery testing:
>
>
>
> INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
>
>
> The other nodes are happy and don’t show this behavior. All the test data
> is
> readable, and everything is fine, but I’m curious why hinted handoff is
> running on one node all the time.
>
>
>
> I searched the bug database, and I found a bug that seems to have the same
> symptoms:
>
> https://issues.apache.org/jira/browse/CASSANDRA-3733
>
> Although it’s been marked fixed in 0.6, this describes my problem exactly.
>
>
>
> I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:
>
>
> http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
>
>
>
> Is anyone else seeing this behavior? What can I do to provide more
> information?
>
>
>
> Steve
>
>
>
>
>
> --
> David Daeschler
>
>
>

Mime
View raw message