cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakub Glapa <jakub.gl...@gmail.com>
Subject Re: Unsuccessful attempt to add a second node to a ring.
Date Fri, 03 Aug 2012 13:38:17 GMT
Hi Roshni,
I was just about to summarize. I got it running.
The problem eventually was the firewall. I didn't have the full control
over the server so I couldn't check the iptables rules.
The incoming traffic on 7000 port was rejected on the second node. I've
just checked one way communication from node2 -> node1 using telnet.
It was very helpful to run cassandra in TRACE mode. It displays a lot of
logs but  was able to compare the behavior with a working ring.
I've noticed that the second node was sending a Gossip SYN message but was
never getting back the SYN_ACK.

Anyway to everybody that will have a problem like that, check your firewall
settings:)

Thanks for help Roshni.

--
regards,
Jakub Glapa


On Fri, Aug 3, 2012 at 9:41 AM, Roshni Rajagopal <
Roshni.Rajagopal@wal-mart.com> wrote:

> Hi Jakub,
>
>  Were you able to resolve the issue?
> For a multi data center setup I do believe some steps are different. You
> may need to set Networktopology as your replication strategy rather than
> simple strategy, and setup a snitch. And mention the rack/dc configurations
> in a config file.
> You can refer to the steps for a multi data center installation.
>
> Regards,
> Roshni
>
>
>
> From: Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Subject: Re: Unsuccessful attempt to add a second node to a ring.
>
> I found a similar thread from March :
> http://www.mail-archive.com/user@cassandra.apache.org/msg21007.html
>
> For me clearing the data and starting from the beginning didn't help.
>
> It's interesting because on my dev environment I was able to add another
> node without any problems.
>
> The only difference is that the second node now is in a different data
> center. (but I'm not using any different settings, SimpleSnitch)
> 7000,9160,7199 ports were open between those 2 nodes.
>
> How else can I check if the communication between those 2 nodes is working?
> In the logs I see that:
> DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642
> OutboundTcpConnection.java (line 206) attempting to connect to
> NODE1/node1.ip
>
> So I assume that the communication is somehow established?
>
>
> --
> regards,
> Jakub Glapa
>
>
> On Wed, Aug 1, 2012 at 11:36 AM, Jakub Glapa <jakub.glapa@gmail.com
> <mailto:jakub.glapa@gmail.com>> wrote:
> yes it's the same
>
>
>
> --
> regards,
> pozdrawiam,
> Jakub Glapa
>
>
> On Wed, Aug 1, 2012 at 11:24 AM, Roshni Rajagopal <
> Roshni.Rajagopal@wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com>>
> wrote:
> Ok, sorry it may not be required,
> I was thinking of a configuration I had done on my local laptop, where I
> had aliased my IP address.
> In that case the directories and jmx port needed to be different.
>
> Cluster name is same right?
>
>
> From: Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com
> ><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org
> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
<
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
> Subject: Re: Unsuccessful attempt to add a second node to a ring.
>
> Hi Roshni,
> no they are the same, my changes in cassandra.yaml were only in the
> listen_address, rpc_address, seeds and initial_token field.
> The rest is exactly the same as on node1.
>
> That's how the file looks on node2:
>
>
>
> cluster_name: 'Test Cluster'
> initial_token: 85070591730234615865843651857942052864
> hinted_handoff_enabled: true
> hinted_handoff_throttle_delay_in_ms: 1
> authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
> authority: org.apache.cassandra.auth.AllowAllAuthority
> partitioner: org.apache.cassandra.dht.RandomPartitioner
> data_file_directories:
>     - /data/servers/cassandra_sbe_edtool/cassandra_data/data
> commitlog_directory:
> /data/servers/cassandra_sbe_edtool/cassandra_data/commitlog
> saved_caches_directory:
> /data/servers/cassandra_sbe_edtool/cassandra_data/saved_caches
> commitlog_sync: periodic
> commitlog_sync_period_in_ms: 10000
> seed_provider:
>     - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>       parameters:
>           - seeds: "NODE1"
> flush_largest_memtables_at: 0.75
> reduce_cache_sizes_at: 0.85
> reduce_cache_capacity_to: 0.6
> concurrent_reads: 32
> concurrent_writes: 32
> memtable_flush_queue_size: 4
> sliced_buffer_size_in_kb: 64
> storage_port: 7000
> ssl_storage_port: 7001
> listen_address: NODE2
> rpc_address: NODE2
> rpc_port: 9160
> rpc_keepalive: true
> rpc_server_type: sync
> thrift_framed_transport_size_in_mb: 15
> thrift_max_message_length_in_mb: 16
> incremental_backups: false
> snapshot_before_compaction: false
> column_index_size_in_kb: 64
> in_memory_compaction_limit_in_mb: 64
> multithreaded_compaction: false
> compaction_throughput_mb_per_sec: 16
> compaction_preheat_key_cache: true
> rpc_timeout_in_ms: 10000
> endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
> dynamic_snitch_update_interval_in_ms: 100
> dynamic_snitch_reset_interval_in_ms: 600000
> dynamic_snitch_badness_threshold: 0.1
> request_scheduler: org.apache.cassandra.scheduler.NoScheduler
> index_interval: 128
> encryption_options:
>     internode_encryption: none
>     keystore: conf/.keystore
>     keystore_password: cassandra
>     truststore: conf/.truststore
>     truststore_password: cassandra
>
>
>
>
> --
> regards,
> pozdrawiam,
> Jakub Glapa
>
>
> On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajagopal <
> Roshni.Rajagopal@wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com
> ><mailto:Roshni.Rajagopal@wal-mart.com<mailto:
> Roshni.Rajagopal@wal-mart.com>>> wrote:
> Jakub,
>
> Have you set the
> Data, commitlog, saved cache directories to different ones in each yaml
> file for each node?
>
> Regards,
> Roshni
>
>
> From: Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com
> ><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>><mailto:
> jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com><mailto:
> jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org
> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org
> >><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org
> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>"
<
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>>
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>>
> Subject: Unsuccessful attempt to add a second node to a ring.
>
> Hi Everybody!
>
> I'm trying to add a second node to an already operating one node cluster.
>
> Some specs:
> - cassandra 1.0.7
> - both nodes have a routable listen_address and rpc_address.
> - Ports are open: (from node2) telnet node1 7000 is successful
> - Seeds parameter on node2 points to node 1.
>
> [node1] nodetool -h localhost ring
> Address         DC          Rack        Status State   Load
>  Owns    Token
> node1.ip     datacenter1 rack1       Up     Normal  74.33 KB
>  100.00% 0
>
> - initial token on node2 was specified
>
> I see something like that in the logs on node2:
>
> DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76)
> collectTimeOrderedData
>  INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667)
> JOINING: waiting for ring and schema information
> DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642
> OutboundTcpConnection.java (line 206) attempting to connect to
> NODE1/node1.ip
> DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java
> (line 86) Disseminating load info ...
>  INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667)
> JOINING: schema complete, ready to bootstrap
> DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ...
> got ring + schema info
>  INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667)
> JOINING: getting bootstrap token
> DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token
> manually specified as 85070591730234615865843651857942052864
> DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying
> mutation of row 4c
>
>
> but it doesn't join the ring:
>
> [node2] nodetool -h localhost ring
> Address         DC          Rack        Status State   Load
>  Owns    Token
> node2.ip   datacenter1 rack1       Up     Normal  13.49 KB        100.00%
> 85070591730234615865843651857942052864
>
>
>
> I'm attaching the full log from node2 startup in debug mode.
>
>
>
> PS.
> When I didn't specified the initial token on node2 I ended up with
> exception like that:
> "Exception encountered during startup: No other nodes seen!  Unable to
> bootstrap.If you intended to start a single-node cluster, you should make
> sure your broadcast_address (or listen_address) is listed as a seed.
> Otherwise, you need to determine why the seed being contacted has no
> knowledge of the rest of the cluster.  Usually, this can be solved by
> giving all nodes the same seed list."
>
>
> I'm not sure how to proceed now. I found a couple of posts with problems
> like that but they weren't very useful.
>
> --
> regards,
> Jakub Glapa
>
> This email and any files transmitted with it are confidential and intended
> solely for the individual or entity to whom they are addressed. If you have
> received this email in error destroy it immediately. *** Walmart
> Confidential ***
>
> This email and any files transmitted with it are confidential and intended
> solely for the individual or entity to whom they are addressed. If you have
> received this email in error destroy it immediately. *** Walmart
> Confidential ***
>
>
> This email and any files transmitted with it are confidential and intended
> solely for the individual or entity to whom they are addressed. If you have
> received this email in error destroy it immediately. *** Walmart
> Confidential ***
>

Mime
View raw message