incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Solovyov <boris.solov...@gmail.com>
Subject Re: Nodetool doesn't shows two nodes
Date Wed, 13 Feb 2013 15:37:55 GMT
Thank you Alain. I will check the things you suggest and report my results.

- Boris


On Wed, Feb 13, 2013 at 7:54 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> Hi Boris.
>
> "I feel like I have made a beginner's mistake"
> That's an horrible feeling :D. I'll try to help ;)
>
> "cluster_name: 'TS'"
> Are you sure you used the same name for both node ?
>
> "I can connect to port 7000"
> You can check all the ports needed there
> http://www.datastax.com/docs/1.2/install/install_ami and open them in
> security group once and for all so you won't be wondering this anymore.
>
> "listen_address: 10.145.232.190"
> "INFO 19:36:32,710 Node /107.22.114.19 state jump to normal"
> There is "10.145.232.190" defined as listen address and you logs says
> that 107.22.114.19 joined the ring and your second ip seems to be
> 23.21.11.193... When you stop an EC2 server, its internal ip may change.
> So I recommend you not to do so, but restart them instead. Anyway you
> should use instance stores and not EBS, and Instance Store can't be stopped
> so you won't have this issue anymore. Don't trust ip-10-145-232-190 which
> is configured at first start in /etc/hostname.
>
> "endpoint_snitch: Ec2MultiRegionSnitch"
> Maybe should you use endpoint_snitch: Ec2Snitch since all your servers are
> in the same zone. You will have to use privates ip everywhere and comment
> the broadcast_address if you do so.
>
>
> The first node has to start with auto_bootsrap: false, while the 2nd one
> could use auto_bootsrap: true. Seeds node must be your first node only, a
> bootstrapping node mustn't be defined as a seed.
>
> "my guess... certainly 30-second timeouts look suspicious"
> This is not a timeout but rather a sleep and it is a normal wait while
> adding a node.
>
> Since your a new user, I guess you have no data. If you want to try some
> conf you can always "reset" your cassandra node by removing .../cassandra/*
> (commitlog, data and saved_caches) after stopping Cassandra.
>
> Good luck with this.
>
>  Alain
>
>
> 2013/2/12 Boris Solovyov <boris.solovyov@gmail.com>
>
>> I've configured 2-node cluster in EC2, key settings as follows:
>>
>> cluster_name: 'TS'
>> num_tokens: 256
>> seed_provider:
>>     - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>>       parameters:
>>           - seeds: "ec2-23-21-11-193.compute-1.amazonaws.com,
>> ec2-107-22-114-19.compute-1.amazonaws.com"
>> listen_address: 10.145.232.190
>> broadcast_address: ec2-23-21-11-193.compute-1.amazonaws.com
>> rpc_address: 0.0.0.0
>>  endpoint_snitch: Ec2MultiRegionSnitch
>>
>> On other node, it is similar, but of course the listen and broadcast
>> address are different. Now, when I start Cassandra, I see in the logs
>>
>> INFO 19:35:32,348 JOINING: waiting for ring information
>>
>> And then after 30 seconds, it says a bunch of things like this:
>>
>> JOINING: schema complete, ready to bootstrap
>> JOINING: getting bootstrap token
>> Enqueuing flush of Memtable...
>> JOINING: sleeping 30000 ms for pending range setup
>> JOINING: Starting to bootstrap...
>> Bootstrap completed! for the tokens [....]
>>
>> Finally, after some more memtable flushing,
>>
>> INFO 19:36:32,710 Node /107.22.114.19 state jump to normal
>> INFO 19:36:32,722 Startup completed! Now serving reads.
>>
>> Now, I start the other node, and I see basically the same thing in the
>> logs.
>>
>> Running nodetool status, I see what looks like two single-node clusters!
>>
>> [root@ip-10-147-171-160 ~]# nodetool status
>> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns   Host ID
>>             Rack
>> UN  107.22.114.19     21 KB      256     100.0%
>>  f7a24bd2-8cb9-499d-806c-d9e548f34b8d  1a
>>
>> [root@ip-10-145-232-190 ~]# nodetool status
>> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns   Host ID
>>             Rack
>> UN  23.21.11.193      21 KB      256     100.0%
>>  9d70f022-03cf-488a-807d-22e991761483  1a
>>
>> It looks to me like nodes didn't communicate with each other like I
>> thought they would, and timed out waiting for gossip to tell them which
>> nodes are in the ring (I'm new to Cassandra, but this is my guess...
>> certainly 30-second timeouts look suspicious). I checked with telnet, and
>> from each node I can connect to port 7000 on the other node (both on
>> internal and public IP). I feel like I have made a beginner's mistake.
>> Anyone has a suggestion where to look next?
>>
>> - Boris
>>
>
>

Mime
View raw message