cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Hinted handoff question
Date Fri, 07 Oct 2011 14:38:48 GMT
This was fixed in 0.7.8 by CASSANDRA-2928

In general it's a good idea to upgrade to the latest stable release
before spending a lot of time debugging :)

On Fri, Oct 7, 2011 at 8:33 AM, Rene Kochen
<Rene.Kochen@emea.schange.com> wrote:
> If I trigger hint delivery using JMX, it works. I see in the log:
>
>
>
> 2011-10-07 15:17:51,216 INFO 15:17:51,216 Started hinted handoff for
> endpoint /172.16.108.19
>
> However, it does not work automatically (i.e. after node 3 is up again).
>
>
>
> I tested it with Cassandra 0.8.6 and there it works as expected.
> Unfortunately, we have 0.7.7. in production.
>
>
>
> Thanks!
>
>
>
> Rene
>
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 22:51
>
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
>
>
> At debug level you should see some log messages such as…
>
>
>
> "Checking remote schema before delivering hints…"
>
> "Sleeping {}ms to stagger hint delivery…"
>
> "Endpoint {} died before hint delivery, aborting…"
>
> "Started hinted handoff for endpoint…"
>
>
>
> If you want to move on, you can deliver the hints using JMX or the node
> should start to deliver them if you restart it (i think).
>
>
>
> Cheers
>
>
>
> -----------------
>
> Aaron Morton
>
> Freelance Cassandra Developer
>
> @aaronmorton
>
> http://www.thelastpickle.com
>
>
>
> On 6/10/2011, at 11:40 PM, Rene Kochen wrote:
>
> Node 3 is up (using ring on node 1).
>
>
>
> There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).
>
>
>
> This is the log from node 1 when nodes 3 starts:
>
>
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted,
> now UP again
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal,
> token 170141183460469000000000000000000000000
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to
> normal
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes
> -> empty pending ranges for Traxis
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to
> /172.16.108.19
>
> 2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
>
> 2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
>
> 2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...
>
>
>
> The countPendingHints on node 1 is {172.16.108.19=5176}
>
>
>
> Thanks
>
>
>
> Rene
>
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 12:13
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
>
>
> Check that node 1 sees node 3 as UP (via ring).
>
>
>
> Check the tpstats on node 1, is there an active HH task ?
>
>
>
> Take another crawl through the logs.
>
>
>
> Cheers
>
>
>
> -----------------
>
> Aaron Morton
>
> Freelance Cassandra Developer
>
> @aaronmorton
>
> http://www.thelastpickle.com
>
>
>
> On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
>
>
> I'am using Cassandra 0.7.7 and have a question about hinted handoff.
>
>
>
> I have a cluster of three nodes.
>
> I stop node 3.
>
> I see that the hint count for node 3 increases on node 1 (countPendingHints
> = 28709).
>
> However, when I start node 3 again, I cannot see anything in the log
> regarding hinted handoff delivery.
>
> If I look at the countPendingHints on node 1, it still says 28709.
>
>
>
> hinted_handoff_enabled is enabled.
>
>
>
> Thanks!
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message