Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D48152A60 for ; Thu, 5 May 2011 18:34:32 +0000 (UTC) Received: (qmail 63716 invoked by uid 500); 5 May 2011 18:34:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 63634 invoked by uid 500); 5 May 2011 18:34:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 63626 invoked by uid 99); 5 May 2011 18:34:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:34:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Len.Bucchino@veritix.com designates 63.251.250.70 as permitted sender) Received: from [63.251.250.70] (HELO mx1.flashseats.com) (63.251.250.70) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 18:34:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=veritix.com; i=Len.Bucchino@veritix.com; q=dns/txt; s=corpdkim; t=1304620465; x=1336156465; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; z=From:=20Len=20Bucchino=20|To: =20"user@cassandra.apache.org"=20|Subject:=20RE:=20New=20node=20not=20joining|Date:=20T hu,=205=20May=202011=2018:31:05=20+0000|Message-ID:=20|References:=20=0D=0A=09=0D=0A=20=0D=0A=20|In-Reply-To:=20|MIME-Version:=20 1.0; bh=aSWeHl929+mqP6ocWsc4cW5pJ7loE3ZjWIss/+osSJ0=; b=mJpyOFeTFBsP3HzEu1ivABoPFs+J36iiKAV27a+RDcyCsBrQRqmh8b95 F1xnR6e/7vcOusaQB+SG8/XDSjvP8dZWN3yZqlKq7OvweGmoKPB/4HUAA 9uaHatSWmrGXvMv; Received: from unknown (HELO webmail101.corp.veritix.com) ([192.168.80.110]) by mx1.flashseats.com with ESMTP; 05 May 2011 11:34:05 -0700 Received: from MAILSTORE101.corp.veritix.com ([::1]) by webmail101.corp.veritix.com ([::1]) with mapi id 14.01.0270.001; Thu, 5 May 2011 11:31:07 -0700 From: Len Bucchino To: "user@cassandra.apache.org" Subject: RE: New node not joining Thread-Topic: New node not joining Thread-Index: AQHMCvRhfRbaVgD6iEWV9TlkbHIn9JR+Pa44gACp7ID//6IegIAABG1w Date: Thu, 5 May 2011 18:31:05 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.1.33] Content-Type: multipart/alternative; boundary="_000_B1B224121353FA42A621A0AC86A3735DE0164BMailStore101corpv_" MIME-Version: 1.0 --_000_B1B224121353FA42A621A0AC86A3735DE0164BMailStore101corpv_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Also, setting auto_bootstrap to false and setting token to the one that it = said it would use in the logs allows the new node to join the ring. From: Len Bucchino [mailto:Len.Bucchino@veritix.com] Sent: Thursday, May 05, 2011 1:25 PM To: user@cassandra.apache.org Subject: RE: New node not joining Adding the fourth node to the cluster with an empty schema using auto_boots= trap was not successful. A nodetool netstats on the new node shows "Mode: = Joining: getting bootstrap token" similar to what the third node did before= it was manually added. Also, there are no exceptions in the logs but it n= ever joins the ring. From: Sanjeev Kulkarni [mailto:sanjeev@locomatix.com] Sent: Thursday, May 05, 2011 11:47 AM To: user@cassandra.apache.org Subject: Re: New node not joining Hi Len, This looks like a decent workaround. I would be very interested to see how = the addition of the 4th node went. Please post it whenever you get a chance= . Thanks! On Thu, May 5, 2011 at 6:47 AM, Len Bucchino > wrote: I have the same problem on 0.7.5 auto bootstrapping a 3rd node onto an empt= y 2 node test cluster (the two nodes were manually added) and the it curren= tly has an empty schema. My log entries look similar to yours. I took the= new token it says its going to use from the log file added it to the yaml = and turned off auto bootstrap and the node added fine. I'm bringing up a 4= th node now and will see if it has the same problem auto bootstrapping. ________________________________ From: Sanjeev Kulkarni [sanjeev@locomatix.com= ] Sent: Thursday, May 05, 2011 2:18 AM To: user@cassandra.apache.org Subject: New node not joining Hey guys, I'm running into what seems like a very basic problem. I have a one node cassandra instance. Version 0.7.5. Freshly installed. Con= tains no data. The cassandra.yaml is the same as the default one that is supplied, except = for data/commitlog/saved_caches directories. I also changed the addresses to point to a externally visible ip address. The cassandra comes up nicely and is ready to accept thrift connections. I do a nodetool and this is what I get. 10.242.217.124 Up Normal 6.54 KB 100.00% 11002286299308678990= 3543147927259579701 Which seems right to me. Now I start another node. Almost identical configuration to the first one. = Except the bootstrap is turned true and seeds appropriately set. When I start the second, I notice that the second one contacts the first no= de to get the new token. I see the following lines in the first machine(the seed machine). INFO [GossipStage:1] 2011-05-05 07:00:20,427 Gossiper.java (line 628) Node = /10.83.111.80 has restarted, now UP again INFO [HintedHandoff:1] 2011-05-05 07:00:55,162 HintedHandOffManager.java (= line 304) Started hinted handoff for endpoint /10.83.111.80 INFO [HintedHandoff:1] 2011-05-05 07:00:55,164 HintedHandOffManager.java (= line 360) Finished hinted hand off of 0 rows to endpoint /10.83.111.80 However when i do a node ring, I still get 10.242.217.124 Up Normal 6.54 KB 100.00% 11002286299308678990= 3543147927259579701 Even though the second node has come up. On the second machine the logs say INFO [main] 2011-05-05 07:00:19,124 StorageService.java (line 504) Joining:= getting load information INFO [main] 2011-05-05 07:00:19,124 StorageLoadBalancer.java (line 351) Sl= eeping 90000 ms to wait for load information... INFO [GossipStage:1] 2011-05-05 07:00:20,828 Gossiper.java (line 628) Node= /10.242.217.124 has restarted, now UP again INFO [HintedHandoff:1] 2011-05-05 07:00:29,548 HintedHandOffManager.java (= line 304) Started hinted handoff for endpoint /10.242.217.124 INFO [HintedHandoff:1] 2011-05-05 07:00:29,550 HintedHandOffManager.java (= line 360) Finished hinted handoff of 0 rows to endpoint /10.242.217.124 INFO [main] 2011-05-05 07:01:49,137 StorageService.java (line 504) Joining= : getting bootstrap token INFO [main] 2011-05-05 07:01:49,148 BootStrapper.java (line 148) New token= will be 24952271262852174037699496069317526837 to assume load from /10.242= .217.124 INFO [main] 2011-05-05 07:01:49,150 Mx4jTool.java (line 72) Will not load = MX4J, mx4j-tools.jar is not in the classpath INFO [main] 2011-05-05 07:01:49,259 CassandraDaemon.java (line 112) Bindin= g thrift service to /10.83.111.80:9160 INFO [main] 2011-05-05 07:01:49,262 CassandraDaemon.java (line 126) Using = TFastFramedTransport with a max frame size of 15728640 bytes. INFO [Thread-5] 2011-05-05 07:01:49,266 CassandraDaemon.java (line 154) Li= stening for thrift clients... This seems to indicate that the second node has joined the ring. And has go= tten its key range. Am I missing anything? Thanks! --_000_B1B224121353FA42A621A0AC86A3735DE0164BMailStore101corpv_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Also, setting auto_bootstrap to false and setting token= to the one that it said it would use in the logs allows the new node to jo= in the ring.

 <= /p>

From: Len Bucc= hino [mailto:Len.Bucchino@veritix.com]
Sent: Thursday, May 05, 2011 1:25 PM
To: user@cassandra.apache.org
Subject: RE: New node not joining

 

Adding the fourth node to= the cluster with an empty schema using auto_bootstrap was not successful.&= nbsp; A nodetool netstats on the new node shows “Mode: Joining: getting bootstrap token” similar to what the third node did before i= t was manually added.  Also, there are no exceptions in the logs but i= t never joins the ring.

 <= /p>

From: Sanjeev = Kulkarni [mailto:sanjeev@locomatix.com]
Sent: Thursday, May 05, 2011 11:47 AM
To: user@cassandra.apache.org
Subject: Re: New node not joining

 

Hi Len,

This looks like a decent workaround. I would be very= interested to see how the addition of the 4th node went. Please post it wh= enever you get a chance.

Thanks!

 

On Thu, May 5, 2011 at 6:47 AM, Len Bucchino <Len.Bucchino@veritix.com> w= rote:

I have the same problem on 0= .7.5 auto bootstrapping a 3rd node onto an empty 2 node test cluster (the t= wo nodes were manually added) and the it currently has an empty schema.  My log entries look similar to yours.  = I took the new token it says its going to use from the log file added it to= the yaml and turned off auto bootstrap and the node added fine.  I'm = bringing up a 4th node now and will see if it has the same problem auto bootstrapping.

 


From: Sanjeev Kulkarni [sanjeev@locomatix.com]=
Sent: Thursday, May 05, 2011 2:18 AM
To: u= ser@cassandra.apache.org
Subject: New node not joining

Hey guys,

I'm running into what se= ems like a very basic problem.

I have a one node cassan= dra instance. Version 0.7.5. Freshly installed. Contains no data.

The cassandra.yaml is th= e same as the default one that is supplied, except for data/commitlog/saved= _caches directories.

I also changed the addre= sses to point to a externally visible ip address.

The cassandra comes up n= icely and is ready to accept thrift connections.

I do a nodetool and this= is what I get.

 =

10.242.217.124  Up =     Normal  6.54 KB         100.00% 1100= 22862993086789903543147927259579701

 =

Which seems right to me.=

 =

Now I start another node= . Almost identical configuration to the first one. Except the bootstrap is = turned true and seeds appropriately set.

When I start the second,= I notice that the second one contacts the first node to get the new token.=

I see the following line= s in the first machine(the seed machine).

 =

INFO [GossipStage:1] 201= 1-05-05 07:00:20,427 Gossiper.java (line 628) Node /10.83.111.80 has restarted, =

now UP again<= /span>

 INFO [HintedHandof= f:1] 2011-05-05 07:00:55,162 HintedHandOffManager.java (line 304) Started h= inted handoff for endpoint /10.83.111.80

 INFO [HintedHandof= f:1] 2011-05-05 07:00:55,164 HintedHandOffManager.java (line 360) Finished = hinted hand

off of 0 rows to endpoin= t /10.83.111.80<= /o:p>

 =

However when i do a node= ring, I still get

 =

10.242.217.124  Up =     Normal  6.54 KB         100.00% 1100= 22862993086789903543147927259579701

 =

Even though the second n= ode has come up. On the second machine the logs say

 =

INFO [main] 2011-05-05 0= 7:00:19,124 StorageService.java (line 504) Joining: getting load informatio= n

 INFO [main] 2011-0= 5-05 07:00:19,124 StorageLoadBalancer.java (line 351) Sleeping 90000 ms to = wait for load information...

 INFO [GossipStage:= 1] 2011-05-05 07:00:20,828 Gossiper.java (line 628) Node /10.242.217.124 has restarted, now UP= again

 INFO [HintedHandof= f:1] 2011-05-05 07:00:29,548 HintedHandOffManager.java (line 304) Started h= inted handoff for endpoint /10.242.217.124

 INFO [HintedHandof= f:1] 2011-05-05 07:00:29,550 HintedHandOffManager.java (line 360) Finished = hinted handoff of 0 rows to endpoint /10.242.217.124

 INFO [main] 2011-0= 5-05 07:01:49,137 StorageService.java (line 504) Joining: getting bootstrap= token

 INFO [main] 2011-0= 5-05 07:01:49,148 BootStrapper.java (line 148) New token will be 2495227126= 2852174037699496069317526837 to assume load from /10.242.217.124

 INFO [main] 2011-0= 5-05 07:01:49,150 Mx4jTool.java (line 72) Will not load MX4J, mx4j-tools.ja= r is not in the classpath

 INFO [main] 2011-0= 5-05 07:01:49,259 CassandraDaemon.java (line 112) Binding thrift service to= /10.83.111.80:9160<= /a>

 INFO [main] 2011-0= 5-05 07:01:49,262 CassandraDaemon.java (line 126) Using TFastFramedTranspor= t with a max frame size of 15728640 bytes.

 INFO [Thread-5] 20= 11-05-05 07:01:49,266 CassandraDaemon.java (line 154) Listening for thrift = clients...

 =

This seems to indicate t= hat the second node has joined the ring. And has gotten its key range. = ;

Am I missing anything?


Thanks!

 =

 

--_000_B1B224121353FA42A621A0AC86A3735DE0164BMailStore101corpv_--