Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 61071 invoked from network); 8 Nov 2010 20:37:55 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 8 Nov 2010 20:37:55 -0000 Received: (qmail 53225 invoked by uid 500); 8 Nov 2010 20:38:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53170 invoked by uid 500); 8 Nov 2010 20:38:25 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53162 invoked by uid 99); 8 Nov 2010 20:38:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 20:38:24 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.161.66] (HELO mail-fx0-f66.google.com) (209.85.161.66) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 20:38:19 +0000 Received: by fxm3 with SMTP id 3so1073393fxm.1 for ; Mon, 08 Nov 2010 12:37:57 -0800 (PST) Received: by 10.223.115.203 with SMTP id j11mr4202187faq.35.1289248677467; Mon, 08 Nov 2010 12:37:57 -0800 (PST) Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com [209.85.160.44]) by mx.google.com with ESMTPS id 15sm213922fal.46.2010.11.08.12.37.54 (version=SSLv3 cipher=RC4-MD5); Mon, 08 Nov 2010 12:37:56 -0800 (PST) Received: by pwj2 with SMTP id 2so163236pwj.31 for ; Mon, 08 Nov 2010 12:37:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.142.230.8 with SMTP id c8mr5387511wfh.229.1289248673453; Mon, 08 Nov 2010 12:37:53 -0800 (PST) Received: by 10.142.232.7 with HTTP; Mon, 8 Nov 2010 12:37:53 -0800 (PST) In-Reply-To: References: <67a5913d-169e-6daa-8483-bac1f184dbf2@me.com> Date: Mon, 8 Nov 2010 21:37:53 +0100 Message-ID: Subject: Re: New nodes won't bootstrap on .66 From: Thibaut Britz To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0cd32eb23556fe04949099e3 --000e0cd32eb23556fe04949099e3 Content-Type: text/plain; charset=ISO-8859-1 I had also multiple keyspaces defined (> 20). All nodes were 64 bit, no mixtures. On Mon, Nov 8, 2010 at 8:23 PM, Dimitry Lvovsky wrote: > We didn't solve it unfortunately and and ended up regenerating the entire > cluster. But, if it helps anyone in the future, we too had multiple > keyspaces when we encountered the problem. > > > > On Mon, Nov 8, 2010 at 5:47 PM, Marc Canaleta wrote: > >> I have just solved the problem removing the second keyspace (manually >> moving its column families to the first). So it seems the problem appears >> when having multiple keyspaces. >> >> 2010/11/8 Thibaut Britz >> >> Hi, >>> >>> No I didn't solve the problem. I reinitialized the cluster and gave each >>> node manually a token before adding data. There are a few messages in >>> multiple threads related to this, so I suspect it's very common and I hope >>> it's gone with 0.7. >>> >>> Thibaut >>> >>> >>> >>> >>> >>> On Sun, Nov 7, 2010 at 6:57 PM, Marc Canaleta wrote: >>> >>>> Hi, >>>> >>>> Did you solve this problem? I'm having the same poblem. I'm trying to >>>> bootstrap a third node in a 0.66 cluster. It has two keyspaces: Keyspace1 >>>> and KeyspaceLogs, both with replication factor 2. >>>> >>>> It starts bootstrapping, receives some streams but it keeps waiting for >>>> streams. I enabled the debug mode. This lines may be useful: >>>> >>>> DEBUG [main] 2010-11-07 17:39:50,052 BootStrapper.java (line 70) >>>> Beginning bootstrap process >>>> DEBUG [main] 2010-11-07 17:39:50,082 StorageService.java (line 160) >>>> Added /10.204.93.16/Keyspace1 as a bootstrap source >>>> ... >>>> DEBUG [main] 2010-11-07 17:39:50,090 StorageService.java (line 160) >>>> Added /10.204.93.16/KeyspaceLogs as a bootstrap source >>>> ... (streaming mesages) >>>> DEBUG [Thread-56] 2010-11-07 17:45:51,706 StorageService.java (line 171) >>>> Removed /10.204.93.16/Keyspace1 as a bootstrap source; remaining is [/ >>>> 10.204.93.16] >>>> ... >>>> (and never ends). >>>> >>>> It seems it is waiting for [/10.204.93.16] when it should be waiting >>>> for /10.204.93.16/KeyspaceLogs. >>>> >>>> The third node is 64 bits, while the two existing nodes are 32 bits. Can >>>> this be a problem? >>>> >>>> Thank you. >>>> >>>> >>>> 2010/10/28 Dimitry Lvovsky >>>> >>>> Maybe your 7000 is being blocked by >>>>> iptables or some firewall or maybe you have it bound ( tag ) >>>>> to localhost instead an ip address. >>>>> >>>>> Hope this helps, >>>>> Dimitry. >>>>> >>>>> >>>>> >>>>> On Thu, Oct 28, 2010 at 5:35 PM, Thibaut Britz < >>>>> thibaut.britz@trendiction.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have the same problem with 0.6.5 >>>>>> >>>>>> New nodes will hang forever in bootstrap mode (no streams are being >>>>>> opened) and the receiver thread just waits for data forever: >>>>>> >>>>>> >>>>>> INFO [Thread-53] 2010-10-27 20:33:37,399 SSTableReader.java (line >>>>>> 120) Sampling index for /hd2/cassandra/data/table_xyz/ >>>>>> table_xyz-3-Data.db >>>>>> INFO [Thread-53] 2010-10-27 20:33:37,444 StreamCompletionHandler.java >>>>>> (line 64) Streaming added /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db >>>>>> >>>>>> Stacktracke: >>>>>> >>>>>> "pool-1-thread-53" prio=10 tid=0x00000000412f2800 nid=0x215c runnable >>>>>> [0x00007fd7cf217000] >>>>>> java.lang.Thread.State: RUNNABLE >>>>>> at java.net.SocketInputStream.socketRead0(Native Method) >>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129) >>>>>> at >>>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >>>>>> at >>>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:258) >>>>>> at >>>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:317) >>>>>> - locked <0x00007fd7e77e0520> (a java.io.BufferedInputStream) >>>>>> at >>>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:126) >>>>>> at >>>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192) >>>>>> at >>>>>> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1154) >>>>>> at >>>>>> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 28, 2010 at 12:44 PM, aaron morton < >>>>>> aaron@thelastpickle.com> wrote: >>>>>> >>>>>>> The best approach is to manually select the tokens, see the Load >>>>>>> Balancing section http://wiki.apache.org/cassandra/Operations Also >>>>>>> >>>>>>> Are there any log messages in the existing nodes or the new one which >>>>>>> mention each other? >>>>>>> >>>>>>> Is this a production system? Is it still running ? >>>>>>> >>>>>>> Sorry there is not a lot to go on, it sounds like you've done the >>>>>>> right thing. I'm assuming things like the Cluster Name, seed list and port >>>>>>> numbers are set correct as the new node got some data. >>>>>>> >>>>>>> You'll need to dig through the logs a bit more to see that the boot >>>>>>> strapping started and what was the last message it logged. >>>>>>> >>>>>>> Good Luck. >>>>>>> Aaron >>>>>>> >>>>>>> On 27 Oct 2010, at 22:40, Dimitry Lvovsky wrote: >>>>>>> >>>>>>> Hi Aaron, >>>>>>> Thanks for your reply. >>>>>>> >>>>>>> We still haven't solved this unfortunately. >>>>>>> >>>>>>> How did you start the bootstrap for the .18 node ? >>>>>>> >>>>>>> >>>>>>> Standard way: we set "AutoBootstrap" to true and added all the >>>>>>> servers from the working ring as seeds. >>>>>>> >>>>>>> >>>>>>>> Was it the .18 or the .17 node you tried to add >>>>>>> >>>>>>> >>>>>>> We first tried adding .17, it streamed for a while, took on a 50GB of >>>>>>> load, stopped streaming but then didn't enter into the ring. We left it for >>>>>>> a few days to see if it would come in, but no luck. After that we did >>>>>>> decommission and removeToken ( in that order) operations. >>>>>>> Since we couldn't get .17 in we tried again with .18. Before doing >>>>>>> so we increased the RpcTimeoutInMillis from 1000, to 10000 having read that >>>>>>> this may cause the problem of nodes not entering into the ring. It's been >>>>>>> going since friday and still, like .17, won't come into the ring. >>>>>>> >>>>>>> Does it have a token in the config or did you use nodetool move to >>>>>>>> set it >>>>>>> >>>>>>> No we didn't manually set the token in the config, rather we were >>>>>>> relaying on the token to be assigned durring bootstrap from the >>>>>>> RandomPartitioner. >>>>>>> >>>>>>> Again thanks for the help. >>>>>>> >>>>>>> Dimitry. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 26, 2010 at 10:14 PM, Aaron Morton < >>>>>>> aaron@thelastpickle.com> wrote: >>>>>>> >>>>>>>> Dimitry, Did you get anywhere with this ? >>>>>>>> >>>>>>>> Was it the .18 or the .17 node you tried to add ? How did you start >>>>>>>> the bootstrap for the .18 node ? Does it have a token in the config or did >>>>>>>> you use nodetool move to set it? >>>>>>>> >>>>>>>> I had a quick look at the code AKAIK the message about removing the >>>>>>>> fat client is logged when the node does not have a record of the token the >>>>>>>> other node as. >>>>>>>> >>>>>>>> Aaron >>>>>>>> >>>>>>>> On 26 Oct, 2010,at 10:42 PM, Dimitry Lvovsky >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> We recently upgraded from .65 to .66 after which we tried adding a >>>>>>>> new node to our cluster. We left it bootstrapping and after 3 days, it still >>>>>>>> refused to join the ring. The strange thing is that nodetool info shows 50GB >>>>>>>> of load and nodetool ring shows that it sees the rest of ring, which it is >>>>>>>> not part of. We tried the process again with another server -- again the >>>>>>>> same thing as before: >>>>>>>> >>>>>>>> >>>>>>>> //from machine 192.168.218 >>>>>>>> >>>>>>>> >>>>>>>> /opt/cassandra/bin/nodetool -h localhost -p 8999 info >>>>>>>> 131373516047318302934572185119435768941 >>>>>>>> Load : 52.85 GB >>>>>>>> Generation No : 1287761987 >>>>>>>> Uptime (seconds) : 323157 >>>>>>>> Heap Memory (MB) : 795.42 / 1945.63 >>>>>>>> >>>>>>>> >>>>>>>> /opt/cassandra/bin/nodetool -h localhost -p 8999 ring >>>>>>>> Address Status Load Range Ring >>>>>>>> 158573510920250391466717289405976537674 >>>>>>>> 192.168.2.22 Up 59.45 GB 28203205416427384773583427414698832202 >>>>>>>> |<--| >>>>>>>> 192.168.2.23 Up 44.95 GB 60562227403709245514637766500430120055 | | >>>>>>>> 192.168.2.20 Up 47.15 GB 104160057322065544623939416372654814065 | | >>>>>>>> 192.168.2.21 Up 61.04 GB 158573510920250391466717289405976537674 >>>>>>>> |-->| >>>>>>>> >>>>>>>> opt/cassandra/bin/nodetool -h localhost -p 8999 streams >>>>>>>> Mode: Bootstrapping >>>>>>>> Not sending any streams. >>>>>>>> Not receiving any streams. >>>>>>>> >>>>>>>> >>>>>>>> Whats more, while looking at the log of one of the nodes I see >>>>>>>> gossip messages from 192.168.1.17 -- the first node we tried to add to the >>>>>>>> cluster but which is not running at the the time of the log message: >>>>>>>> INFO [Timer-0] 2010-10-26 02:13:20,340 Gossiper.java (line 406) >>>>>>>> FatClient /192.168.2.17 has been silent for 3600000ms, removing >>>>>>>> from gossip >>>>>>>> INFO [GMFD:1] 2010-10-26 02:13:51,398 Gossiper.java (line 591) Node >>>>>>>> /192.168.2.17 is now part of the cluster >>>>>>>> >>>>>>>> >>>>>>>> Thanks in advance for the help, >>>>>>>> Dimitry >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dimitry Lvovsky >>>>>>> Director of Engineering >>>>>>> ReviewPro >>>>>>> www.reviewpro.com >>>>>>> +34 616 337 103 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Dimitry Lvovsky >>>>> Director of Engineering >>>>> ReviewPro >>>>> www.reviewpro.com >>>>> +34 616 337 103 >>>>> >>>> >>>> >>> >> > > > -- > Dimitry Lvovsky > Director of Engineering > ReviewPro > www.reviewpro.com > +34 616 337 103 > --000e0cd32eb23556fe04949099e3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I had also multiple keyspaces defined (> 20). All nodes were 64 bit, no = mixtures.


On Mon, Nov 8, 2010 at 8:23= PM, Dimitry Lvovsky <dimitry@reviewpro.com> wrote:
We didn't sol= ve it=A0unfortunately=A0and and ended up regenerating the entire cluster. = =A0But, if it helps anyone in the future, we too had multiple keyspaces whe= n we encountered the problem. =A0



On Mon, Nov 8, 2010 at 5:47 PM, Marc Canaleta <mcanaleta@gmail.com&g= t; wrote:
I have just solved the problem removing the second keyspace (manually movin= g its column families to the first). So it seems the problem appears when h= aving multiple keyspaces.

2010/11/8 Thiba= ut Britz <thibaut.britz@trendiction.com>

Hi,

No I d= idn't solve the problem. I reinitialized the cluster and gave each node= manually a token before adding data. There are a few messages in multiple = threads related to this, so I suspect it's very common and I hope it= 9;s gone with 0.7.

Thibaut





On Sun, Nov 7, 2010 at 6:57 PM, Marc Canaleta = <mcanaleta@gmai= l.com> wrote:
Hi,

Did you solve this problem? I'm having the same = poblem. I'm trying to bootstrap a third node in a 0.66 cluster. It has = two keyspaces: Keyspace1 and KeyspaceLogs, both with replication factor 2.= =A0

It starts bootstrapping, receives some streams but it k= eeps waiting for streams. I enabled the debug mode. This lines may be usefu= l:

DEBUG [main] 2010-11-07 17:39:50,052 BootS= trapper.java (line 70) Beginning bootstrap process
DEBUG [main] 2010-11-07 17:39:50,082 StorageService.java (line 16= 0) Added /10.20= 4.93.16/Keyspace1 as a bootstrap source
...
DEBUG [main] 2010-11-07 17:39:50,090 StorageService.java (line 160) Ad= ded /10.204.= 93.16/KeyspaceLogs as a bootstrap source
... (streaming mesages)
DEBUG [Thread-56] 2010-11= -07 17:45:51,706 StorageService.java (line 171) Removed /10.204.93.16/Keyspace1 as a b= ootstrap source; remaining is [/10.204.93.16]
...
(and never ends).

It seem= s it is waiting for=A0=A0[/10.204.93.16] when it should be waiting for /10.204.93.16/KeyspaceLogs.

The third node is 64 bits, while the two existing nodes= are 32 bits. Can this be a problem?

Thank you.


2010/10/28 Dimitry Lvovsky= <dimitry@reviewpro.com>

Maybe your =A0=A0= =A0<StoragePort>7000</StoragePort> is being blocked by iptables= or some firewall or maybe you have it bound (<ListenAddress> tag ) = =A0to localhost instead an ip address.

Hope this helps,
Dimitry.



On Thu, Oct 28, 2010 at 5:35 PM, Thibaut Britz <= thibaut.britz@trendiction.com> wrote:
Hi,

I have= the same problem with 0.6.5

New nodes will hang forever in bootstra= p mode (no streams are being opened) and the receiver thread just waits for= data forever:


=A0INFO [Thread-53] 2010-10-27 20:33:37,399 SSTableReader.java (lin= e 120) Sampling index for /hd2/cassandra/data/table_xyz/
table_xyz-3-Data.db
=A0INFO [Thread-53] 2010-10-27 20:33:37,444 StreamCompletionHandler.java (l= ine 64) Streaming added /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db
Stacktracke:

"pool-1-thread-53" prio=3D10 tid=3D0x00= 000000412f2800 nid=3D0x215c runnable [0x00007fd7cf217000]
=A0=A0 java.lang.Thread.State: RUNNABLE
=A0=A0=A0=A0=A0=A0=A0 at java.ne= t.SocketInputStream.socketRead0(Native Method)
=A0=A0=A0=A0=A0=A0=A0 at = java.net.SocketInputStream.read(SocketInputStream.java:129)
=A0=A0=A0=A0= =A0=A0=A0 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)=
=A0=A0=A0=A0=A0=A0=A0 at java.io.BufferedInputStream.read1(BufferedInputStr= eam.java:258)
=A0=A0=A0=A0=A0=A0=A0 at java.io.BufferedInputStream.read(= BufferedInputStream.java:317)
=A0=A0=A0=A0=A0=A0=A0 - locked <0x00007= fd7e77e0520> (a java.io.BufferedInputStream)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.transport.TIOStreamTransport.rea= d(TIOStreamTransport.java:126)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrif= t.transport.TTransport.readAll(TTransport.java:84)
=A0=A0=A0=A0=A0=A0=A0= at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java= :314)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.protocol.TBinaryProtocol.readI32= (TBinaryProtocol.java:262)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.pr= otocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
=A0=A0= =A0=A0=A0=A0=A0 at org.apache.cassandra.thrift.Cassandra$Processor.process(= Cassandra.java:1154)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.thrift.CustomTThreadPoolServe= r$WorkerProcess.run(CustomTThreadPoolServer.java:167)
=A0=A0=A0=A0=A0=A0= =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe= cutor.java:886)
=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPool= Executor$Worker.run(ThreadPoolExecutor.java:908)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)


<= /div>










On Thu, Oct 28, 2010 at 12:44 PM, aaron morton <a= aron@thelastpickle.com> wrote:
The best approach is to manually select the tokens, se= e the Load Balancing section=A0http://wiki.apache.org/cassandra/Operations Also=A0

Are there any log messages in the existing nodes or the new = one which mention each other?=A0

Is this a product= ion system? Is it still running ?=A0

Sorry there i= s not a lot to go on, it sounds like you've done the right thing. I'= ;m assuming things like the Cluster Name, seed list and port numbers are se= t correct as the new node got some data.

You'll need to dig through the logs a bit more to s= ee that the boot strapping started and what was the last message it logged.= =A0

Good Luck.=A0
Aaron

On 27 Oct 2010, at 22:40, Dimitry Lvovsky wrote:

Hi Aaron,
Thanks for your reply.

We still=A0haven't=A0solved this=A0unfortunately.=A0

How did you start the bootstrap for the .18 node ?=A0

=
Standard way:=A0we set "AutoBootstrap" to true and add= ed all the servers from the working ring as seeds. =A0
=A0
Was it the .18 or the .17 node you tried to add

=
We first tried adding .17, it streamed for a while, took on a 50GB of = load, stopped streaming but then didn't enter into the ring. =A0We left= it for a few days to see if it would come in, but no luck. =A0After that w= e did =A0decommission and =A0removeToken ( in that order) operations.=A0
Since we couldn't get .17 in we tried again with .18. =A0Before do= ing so we increased the RpcTimeoutInMillis from 1000, to 10000 having read = that this may cause the problem of nodes not entering into the ring. =A0=A0= It's been going since friday and still, like .17, won't come into t= he ring.

Does it have a token in the config or did you use nodetool move to set it
No we didn't manually set the token in the config, rath= er we were relaying on the token to be assigned durring bootstrap from the= =A0RandomPar= titioner.

Again thanks for the help.

Dim= itry.
=A0=A0

=A0
On Tue, Oct 26, 2010 at 10:14 PM, Aaron Morton &= lt;aaron@thela= stpickle.com> wrote:
Dimitry= , Did you get anywhere with this ?=A0

Was it the .= 18 or the .17 node you tried to add ? How did you start the bootstrap for t= he .18 node ? Does it have a token in the config or did you use nodetool mo= ve to set it?=A0

I had a quick look at the code AKAIK =A0the message abo= ut removing the fat client is logged when the node does not have a record o= f the token the other node as.=A0

Aaron

On 26 Oct, 2010,at 10:42 PM, Dimitry Lvovsky <dimitry@reviewpro.com> wro= te:

Hi All,
We recently upgraded from .65 to .66 after which we tried adding a new node= to our cluster. We left it bootstrapping and after 3 days, it still refus= ed to join the ring. The strange thing is that nodetool info shows 50GB = of load and nodetool ring shows that it sees the rest of ring, which it is= not part of. We tried the process again with another server -- again the = same thing as before:


//from machine 192.168.218


/opt/cassandra/bin/nodetool -h localhost -p 8999 info
131373516047318302934572185119435768941
Load : 52.85 GB
Generation No : 1287761987
Uptime (seconds) : 323157
Heap Memory (MB) : 795.42 / 1945.63


/opt/cassandra/bin/nodetool -h localhost -p 8999 ring
Address Status Load Range = Ring
158573510920250391466717289405976537= 674
192.168.2.22 Up 59.45 GB 282032054164273847735834274146988322= 02 |<--|
192.168.2.23 Up 44.95 GB 605622274037092455146377665004301200= 55 | |
192.168.2.20 Up 47.15 GB 104160057322065544623939416372654814= 065 | |
192.168.2.21 Up 61.04 GB 158573510920250391466717289405976537= 674 |-->|

opt/cassandra/bin/nodetool -h localhost -p 8999 streams
Mode: Bootstrapping
Not sending any streams.
Not receiving any streams.


Whats more, while looking at the log of one of the nodes I see gossip messa= ges from 192.168.1.17 -- the first node we tried to add to the cluster but = which is not running at the the time of the log message:
INFO [Timer-0] 2010-10-26 02:13:20,340 Gossiper.java (line 406) FatClient = /192.168.2.17 has be= en silent for 3600000ms, removing from gossip
INFO [GMFD:1] 2010-10-26 02:13:51,398 Gossiper.java (line 591) Node /192.168.2.17 is now part = of the cluster


Thanks in advance for the help,
Dimitry

=

--
Dimitry Lvovsky
Director of Engineering
= ReviewPro
www.re= viewpro.com
+34 616 337 103





--
Dimitry Lvo= vsky
Director of Engineering
ReviewPro
www.reviewpro.com
+34 616 337 103






--
Dimitry Lvovsky
Dire= ctor of Engineering
ReviewPro
www.reviewpro.com
+34 616 337 103

--000e0cd32eb23556fe04949099e3--