From user-return-16467-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu May 5 13:50:35 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5BEFE2F2F for ; Thu, 5 May 2011 13:50:35 +0000 (UTC) Received: (qmail 34548 invoked by uid 500); 5 May 2011 13:50:33 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 34525 invoked by uid 500); 5 May 2011 13:50:32 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 34517 invoked by uid 99); 5 May 2011 13:50:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 13:50:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Len.Bucchino@veritix.com designates 63.251.250.70 as permitted sender) Received: from [63.251.250.70] (HELO mx1.flashseats.com) (63.251.250.70) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 13:50:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=veritix.com; i=Len.Bucchino@veritix.com; q=dns/txt; s=corpdkim; t=1304603427; x=1336139427; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; z=From:=20Len=20Bucchino=20|To: =20"user@cassandra.apache.org"=20|Subject:=20RE:=20New=20node=20not=20joining|Date:=20T hu,=205=20May=202011=2013:47:07=20+0000|Message-ID:=20|References:=20|In-Reply-To:=20|MIME-Version:=201.0; bh=7unSVQfYTatAS/Mr0dZiHS4QoBIBREVdgD0hmf+57tk=; b=FiZZ3G6qPTXMJLgHViJ6I0gRsv2Y2r/rEP3f8EFQqG7INPTWKIDecVUI nNRz8hHylsh1KSgVY5JLHJqgGgIG6BLqSY0txrVOL9gFfWoYvtdWty5Fu kFelIh2kTbXoJE0; Received: from unknown (HELO webmail101.corp.veritix.com) ([192.168.80.110]) by mx1.flashseats.com with ESMTP; 05 May 2011 06:50:06 -0700 Received: from MAILSTORE101.corp.veritix.com ([::1]) by webmail101.corp.veritix.com ([::1]) with mapi id 14.01.0270.001; Thu, 5 May 2011 06:47:08 -0700 From: Len Bucchino To: "user@cassandra.apache.org" Subject: RE: New node not joining Thread-Topic: New node not joining Thread-Index: AQHMCvRhfRbaVgD6iEWV9TlkbHIn9JR+Pa44 Date: Thu, 5 May 2011 13:47:07 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [75.17.233.227] Content-Type: multipart/alternative; boundary="_000_B1B224121353FA42A621A0AC86A3735DE00CB4MailStore101corpv_" MIME-Version: 1.0 --_000_B1B224121353FA42A621A0AC86A3735DE00CB4MailStore101corpv_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I have the same problem on 0.7.5 auto bootstrapping a 3rd node onto an empt= y 2 node test cluster (the two nodes were manually added) and the it curren= tly has an empty schema. My log entries look similar to yours. I took the= new token it says its going to use from the log file added it to the yaml = and turned off auto bootstrap and the node added fine. I'm bringing up a 4= th node now and will see if it has the same problem auto bootstrapping. ________________________________ From: Sanjeev Kulkarni [sanjeev@locomatix.com] Sent: Thursday, May 05, 2011 2:18 AM To: user@cassandra.apache.org Subject: New node not joining Hey guys, I'm running into what seems like a very basic problem. I have a one node cassandra instance. Version 0.7.5. Freshly installed. Con= tains no data. The cassandra.yaml is the same as the default one that is supplied, except = for data/commitlog/saved_caches directories. I also changed the addresses to point to a externally visible ip address. The cassandra comes up nicely and is ready to accept thrift connections. I do a nodetool and this is what I get. 10.242.217.124 Up Normal 6.54 KB 100.00% 11002286299308678990= 3543147927259579701 Which seems right to me. Now I start another node. Almost identical configuration to the first one. = Except the bootstrap is turned true and seeds appropriately set. When I start the second, I notice that the second one contacts the first no= de to get the new token. I see the following lines in the first machine(the seed machine). INFO [GossipStage:1] 2011-05-05 07:00:20,427 Gossiper.java (line 628) Node = /10.83.111.80 has restarted, now UP again INFO [HintedHandoff:1] 2011-05-05 07:00:55,162 HintedHandOffManager.java (= line 304) Started hinted handoff for endpoint /10.83.111.80 INFO [HintedHandoff:1] 2011-05-05 07:00:55,164 HintedHandOffManager.java (= line 360) Finished hinted hand off of 0 rows to endpoint /10.83.111.80 However when i do a node ring, I still get 10.242.217.124 Up Normal 6.54 KB 100.00% 11002286299308678990= 3543147927259579701 Even though the second node has come up. On the second machine the logs say INFO [main] 2011-05-05 07:00:19,124 StorageService.java (line 504) Joining:= getting load information INFO [main] 2011-05-05 07:00:19,124 StorageLoadBalancer.java (line 351) Sl= eeping 90000 ms to wait for load information... INFO [GossipStage:1] 2011-05-05 07:00:20,828 Gossiper.java (line 628) Node= /10.242.217.124 has restarted, now UP again INFO [HintedHandoff:1] 2011-05-05 07:00:29,548 HintedHandOffManager.java (= line 304) Started hinted handoff for endpoint /10.242.217.124 INFO [HintedHandoff:1] 2011-05-05 07:00:29,550 HintedHandOffManager.java (= line 360) Finished hinted handoff of 0 rows to endpoint /10.242.217.124 INFO [main] 2011-05-05 07:01:49,137 StorageService.java (line 504) Joining= : getting bootstrap token INFO [main] 2011-05-05 07:01:49,148 BootStrapper.java (line 148) New token= will be 24952271262852174037699496069317526837 to assume load from /10.242= .217.124 INFO [main] 2011-05-05 07:01:49,150 Mx4jTool.java (line 72) Will not load = MX4J, mx4j-tools.jar is not in the classpath INFO [main] 2011-05-05 07:01:49,259 CassandraDaemon.java (line 112) Bindin= g thrift service to /10.83.111.80:9160 INFO [main] 2011-05-05 07:01:49,262 CassandraDaemon.java (line 126) Using = TFastFramedTransport with a max frame size of 15728640 bytes. INFO [Thread-5] 2011-05-05 07:01:49,266 CassandraDaemon.java (line 154) Li= stening for thrift clients... This seems to indicate that the second node has joined the ring. And has go= tten its key range. Am I missing anything? Thanks! --_000_B1B224121353FA42A621A0AC86A3735DE00CB4MailStore101corpv_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I have the same problem on 0.7.5 auto bootstrapping a 3rd node onto = an empty 2 node test cluster (the two nodes were manually added) and the it= currently has an empty schema.  My log entries look similar to yours.  I took the new token it= says its going to use from the log file added it to the yaml and turned of= f auto bootstrap and the node added fine.  I'm bringing up a 4th node = now and will see if it has the same problem auto bootstrapping.

From: Sanjeev Kulkarni [sanjeev@locomati= x.com]
Sent: Thursday, May 05, 2011 2:18 AM
To: user@cassandra.apache.org
Subject: New node not joining

Hey guys,
I'm running into what seems like a very basic problem.
I have a one node cassandra instance. Version 0.7.5. Freshly installed= . Contains no data.
The cassandra.yaml is the same as the default one that is supplied, ex= cept for data/commitlog/saved_caches directories.
I also changed the addresses to point to a externally visible ip addre= ss.
The cassandra comes up nicely and is ready to accept thrift connection= s.
I do a nodetool and this is what I get.

10.242.217.124  Up     Normal  6.54 KB   &nbs= p;     100.00% 110022862993086789903543147927259579701

Which seems right to me.

Now I start another node. Almost identical configuration to the first = one. Except the bootstrap is turned true and seeds appropriately set.
When I start the second, I notice that the second one contacts the fir= st node to get the new token.
I see the following lines in the first machine(the seed machine).

INFO [GossipStage:1] 2011-05-05 07:00:20,427 Gossiper.java (line 628) = Node /10.83.111.80 ha= s restarted, 
now UP again
 INFO [HintedHandoff:1] 2011-05-05 07:00:55,162 HintedHandOffMana= ger.java (line 304) Started hinted handoff for endpoint /10.83.111.80
 INFO [HintedHandoff:1] 2011-05-05 07:00:55,164 HintedHandOffMana= ger.java (line 360) Finished hinted hand
off of 0 rows to endpoint /10.83.111.80

However when i do a node ring, I still get

10.242.217.124  Up     Normal  6.54 KB   &nbs= p;     100.00% 110022862993086789903543147927259579701

Even though the second node has come up. On the second machine the log= s say

INFO [main] 2011-05-05 07:00:19,124 StorageService.java (line 504) Joi= ning: getting load information
 INFO [main] 2011-05-05 07:00:19,124 StorageLoadBalancer.java (li= ne 351) Sleeping 90000 ms to wait for load information...
 INFO [GossipStage:1] 2011-05-05 07:00:20,828 Gossiper.java (line= 628) Node /10.242.217.= 124 has restarted, now UP again
 INFO [HintedHandoff:1] 2011-05-05 07:00:29,548 HintedHandOffMana= ger.java (line 304) Started hinted handoff for endpoint /10.242.217.124
 INFO [HintedHandoff:1] 2011-05-05 07:00:29,550 HintedHandOffMana= ger.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.242.217.124
 INFO [main] 2011-05-05 07:01:49,137 StorageService.java (line 50= 4) Joining: getting bootstrap token
 INFO [main] 2011-05-05 07:01:49,148 BootStrapper.java (line 148)= New token will be 24952271262852174037699496069317526837 to assume load fr= om /10.242.217.124<= /div>
 INFO [main] 2011-05-05 07:01:49,150 Mx4jTool.java (line 72) Will= not load MX4J, mx4j-tools.jar is not in the classpath
 INFO [main] 2011-05-05 07:01:49,259 CassandraDaemon.java (line 1= 12) Binding thrift service to /10.83.111.80:9160
 INFO [main] 2011-05-05 07:01:49,262 CassandraDaemon.java (line 1= 26) Using TFastFramedTransport with a max frame size of 15728640 bytes.
 INFO [Thread-5] 2011-05-05 07:01:49,266 CassandraDaemon.java (li= ne 154) Listening for thrift clients...

This seems to indicate that the second node has joined the ring. And h= as gotten its key range. 
Am I missing anything?

Thanks!

--_000_B1B224121353FA42A621A0AC86A3735DE00CB4MailStore101corpv_--