cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rene Kochen <Rene.Koc...@emea.schange.com>
Subject RE: Hinted handoff question
Date Fri, 07 Oct 2011 13:33:18 GMT
If I trigger hint delivery using JMX, it works. I see in the log:

2011-10-07 15:17:51,216 INFO 15:17:51,216 Started hinted handoff for endpoint /172.16.108.19
However, it does not work automatically (i.e. after node 3 is up again).

I tested it with Cassandra 0.8.6 and there it works as expected. Unfortunately, we have 0.7.7.
in production.

Thanks!

Rene

From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: donderdag 6 oktober 2011 22:51
To: user@cassandra.apache.org
Subject: Re: Hinted handoff question

At debug level you should see some log messages such as...

"Checking remote schema before delivering hints..."
"Sleeping {}ms to stagger hint delivery..."
"Endpoint {} died before hint delivery, aborting..."
"Started hinted handoff for endpoint..."

If you want to move on, you can deliver the hints using JMX or the node should start to deliver
them if you restart it (i think).

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 6/10/2011, at 11:40 PM, Rene Kochen wrote:


Node 3 is up (using ring on node 1).

There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).

This is the log from node 1 when nodes 3 starts:

2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted, now UP again
2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal, token 170141183460469000000000000000000000000
2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to normal
2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes -> empty pending
ranges for Traxis
2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to /172.16.108.19
2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...

The countPendingHints on node 1 is {172.16.108.19=5176}

Thanks

Rene

From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: donderdag 6 oktober 2011 12:13
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Hinted handoff question

Check that node 1 sees node 3 as UP (via ring).

Check the tpstats on node 1, is there an active HH task ?

Take another crawl through the logs.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 6/10/2011, at 10:35 PM, Rene Kochen wrote:



I'am using Cassandra 0.7.7 and have a question about hinted handoff.

I have a cluster of three nodes.
I stop node 3.
I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709).
However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff
delivery.
If I look at the countPendingHints on node 1, it still says 28709.

hinted_handoff_enabled is enabled.

Thanks!


Mime
View raw message