Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8136ADA4A for ; Fri, 3 Aug 2012 13:38:46 +0000 (UTC) Received: (qmail 53607 invoked by uid 500); 3 Aug 2012 13:38:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53467 invoked by uid 500); 3 Aug 2012 13:38:43 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53455 invoked by uid 99); 3 Aug 2012 13:38:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Aug 2012 13:38:42 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jakub.glapa@gmail.com designates 209.85.216.51 as permitted sender) Received: from [209.85.216.51] (HELO mail-qa0-f51.google.com) (209.85.216.51) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Aug 2012 13:38:38 +0000 Received: by qaea16 with SMTP id a16so1996213qae.10 for ; Fri, 03 Aug 2012 06:38:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=iTy2gE3vtZEQ8l6e81OClS7EHNpVyJTwVDcRtplP/+k=; b=YBkqJOudbjgm6PI0T1fIEuYGu/txnvDPjGZQ2Th9LxDwsdTYC9dw9mqqaS0LZDJ6Fh CgloH5LLZGkxqT0GflTVJ60DEoHFs7lo9nDjVgOQqF+Z8B/sYIlY16+pBKklSDOm/HIZ p8Q7DGS9yZmCFFDChwwA4dPdIPFqqSwnS/t7CLyoCF1lYpk739TEPdmydT3MkXpXWleM N3KmdJzuymLlOqvMZV51sLOHLeufQ0u45DyABhxDMw4ryQOX4OKDD8PNglMZtK8592hg C1B/3yD0/6P7AAeHTtNaugNja9nZ/+A5P21YD4G8n/4Yss9IZ7coutKcNIDQ1/CZvEdN 2N0g== MIME-Version: 1.0 Received: by 10.182.0.13 with SMTP id 13mr4676032oba.55.1344001097319; Fri, 03 Aug 2012 06:38:17 -0700 (PDT) Received: by 10.182.50.234 with HTTP; Fri, 3 Aug 2012 06:38:17 -0700 (PDT) In-Reply-To: References: Date: Fri, 3 Aug 2012 14:38:17 +0100 Message-ID: Subject: Re: Unsuccessful attempt to add a second node to a ring. From: Jakub Glapa To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d043c8024fbc40e04c65ca3f6 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043c8024fbc40e04c65ca3f6 Content-Type: text/plain; charset=UTF-8 Hi Roshni, I was just about to summarize. I got it running. The problem eventually was the firewall. I didn't have the full control over the server so I couldn't check the iptables rules. The incoming traffic on 7000 port was rejected on the second node. I've just checked one way communication from node2 -> node1 using telnet. It was very helpful to run cassandra in TRACE mode. It displays a lot of logs but was able to compare the behavior with a working ring. I've noticed that the second node was sending a Gossip SYN message but was never getting back the SYN_ACK. Anyway to everybody that will have a problem like that, check your firewall settings:) Thanks for help Roshni. -- regards, Jakub Glapa On Fri, Aug 3, 2012 at 9:41 AM, Roshni Rajagopal < Roshni.Rajagopal@wal-mart.com> wrote: > Hi Jakub, > > Were you able to resolve the issue? > For a multi data center setup I do believe some steps are different. You > may need to set Networktopology as your replication strategy rather than > simple strategy, and setup a snitch. And mention the rack/dc configurations > in a config file. > You can refer to the steps for a multi data center installation. > > Regards, > Roshni > > > > From: Jakub Glapa > > Reply-To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > Subject: Re: Unsuccessful attempt to add a second node to a ring. > > I found a similar thread from March : > http://www.mail-archive.com/user@cassandra.apache.org/msg21007.html > > For me clearing the data and starting from the beginning didn't help. > > It's interesting because on my dev environment I was able to add another > node without any problems. > > The only difference is that the second node now is in a different data > center. (but I'm not using any different settings, SimpleSnitch) > 7000,9160,7199 ports were open between those 2 nodes. > > How else can I check if the communication between those 2 nodes is working? > In the logs I see that: > DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 > OutboundTcpConnection.java (line 206) attempting to connect to > NODE1/node1.ip > > So I assume that the communication is somehow established? > > > -- > regards, > Jakub Glapa > > > On Wed, Aug 1, 2012 at 11:36 AM, Jakub Glapa > wrote: > yes it's the same > > > > -- > regards, > pozdrawiam, > Jakub Glapa > > > On Wed, Aug 1, 2012 at 11:24 AM, Roshni Rajagopal < > Roshni.Rajagopal@wal-mart.com> > wrote: > Ok, sorry it may not be required, > I was thinking of a configuration I had done on my local laptop, where I > had aliased my IP address. > In that case the directories and jmx port needed to be different. > > Cluster name is same right? > > > From: Jakub Glapa >>> > Reply-To: "user@cassandra.apache.org >>" < > user@cassandra.apache.org user@cassandra.apache.org>> > To: "user@cassandra.apache.org user@cassandra.apache.org>" < > user@cassandra.apache.org user@cassandra.apache.org>> > Subject: Re: Unsuccessful attempt to add a second node to a ring. > > Hi Roshni, > no they are the same, my changes in cassandra.yaml were only in the > listen_address, rpc_address, seeds and initial_token field. > The rest is exactly the same as on node1. > > That's how the file looks on node2: > > > > cluster_name: 'Test Cluster' > initial_token: 85070591730234615865843651857942052864 > hinted_handoff_enabled: true > hinted_handoff_throttle_delay_in_ms: 1 > authenticator: org.apache.cassandra.auth.AllowAllAuthenticator > authority: org.apache.cassandra.auth.AllowAllAuthority > partitioner: org.apache.cassandra.dht.RandomPartitioner > data_file_directories: > - /data/servers/cassandra_sbe_edtool/cassandra_data/data > commitlog_directory: > /data/servers/cassandra_sbe_edtool/cassandra_data/commitlog > saved_caches_directory: > /data/servers/cassandra_sbe_edtool/cassandra_data/saved_caches > commitlog_sync: periodic > commitlog_sync_period_in_ms: 10000 > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "NODE1" > flush_largest_memtables_at: 0.75 > reduce_cache_sizes_at: 0.85 > reduce_cache_capacity_to: 0.6 > concurrent_reads: 32 > concurrent_writes: 32 > memtable_flush_queue_size: 4 > sliced_buffer_size_in_kb: 64 > storage_port: 7000 > ssl_storage_port: 7001 > listen_address: NODE2 > rpc_address: NODE2 > rpc_port: 9160 > rpc_keepalive: true > rpc_server_type: sync > thrift_framed_transport_size_in_mb: 15 > thrift_max_message_length_in_mb: 16 > incremental_backups: false > snapshot_before_compaction: false > column_index_size_in_kb: 64 > in_memory_compaction_limit_in_mb: 64 > multithreaded_compaction: false > compaction_throughput_mb_per_sec: 16 > compaction_preheat_key_cache: true > rpc_timeout_in_ms: 10000 > endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch > dynamic_snitch_update_interval_in_ms: 100 > dynamic_snitch_reset_interval_in_ms: 600000 > dynamic_snitch_badness_threshold: 0.1 > request_scheduler: org.apache.cassandra.scheduler.NoScheduler > index_interval: 128 > encryption_options: > internode_encryption: none > keystore: conf/.keystore > keystore_password: cassandra > truststore: conf/.truststore > truststore_password: cassandra > > > > > -- > regards, > pozdrawiam, > Jakub Glapa > > > On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajagopal < > Roshni.Rajagopal@wal-mart.com > Roshni.Rajagopal@wal-mart.com>>> wrote: > Jakub, > > Have you set the > Data, commitlog, saved cache directories to different ones in each yaml > file for each node? > > Regards, > Roshni > > > From: Jakub Glapa >> jakub.glapa@gmail.com jakub.glapa@gmail.com>>> > Reply-To: "user@cassandra.apache.org > >> >>>" < > user@cassandra.apache.org user@cassandra.apache.org> user@cassandra.apache.org user@cassandra.apache.org>>> > To: "user@cassandra.apache.org user@cassandra.apache.org> user@cassandra.apache.org user@cassandra.apache.org>>" < > user@cassandra.apache.org user@cassandra.apache.org> user@cassandra.apache.org user@cassandra.apache.org>>> > Subject: Unsuccessful attempt to add a second node to a ring. > > Hi Everybody! > > I'm trying to add a second node to an already operating one node cluster. > > Some specs: > - cassandra 1.0.7 > - both nodes have a routable listen_address and rpc_address. > - Ports are open: (from node2) telnet node1 7000 is successful > - Seeds parameter on node2 points to node 1. > > [node1] nodetool -h localhost ring > Address DC Rack Status State Load > Owns Token > node1.ip datacenter1 rack1 Up Normal 74.33 KB > 100.00% 0 > > - initial token on node2 was specified > > I see something like that in the logs on node2: > > DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76) > collectTimeOrderedData > INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667) > JOINING: waiting for ring and schema information > DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 > OutboundTcpConnection.java (line 206) attempting to connect to > NODE1/node1.ip > DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java > (line 86) Disseminating load info ... > INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667) > JOINING: schema complete, ready to bootstrap > DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ... > got ring + schema info > INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667) > JOINING: getting bootstrap token > DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token > manually specified as 85070591730234615865843651857942052864 > DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying > mutation of row 4c > > > but it doesn't join the ring: > > [node2] nodetool -h localhost ring > Address DC Rack Status State Load > Owns Token > node2.ip datacenter1 rack1 Up Normal 13.49 KB 100.00% > 85070591730234615865843651857942052864 > > > > I'm attaching the full log from node2 startup in debug mode. > > > > PS. > When I didn't specified the initial token on node2 I ended up with > exception like that: > "Exception encountered during startup: No other nodes seen! Unable to > bootstrap.If you intended to start a single-node cluster, you should make > sure your broadcast_address (or listen_address) is listed as a seed. > Otherwise, you need to determine why the seed being contacted has no > knowledge of the rest of the cluster. Usually, this can be solved by > giving all nodes the same seed list." > > > I'm not sure how to proceed now. I found a couple of posts with problems > like that but they weren't very useful. > > -- > regards, > Jakub Glapa > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** > > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** > --f46d043c8024fbc40e04c65ca3f6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Roshni,=C2=A0
I was just about to summarize. I got it running.
=
The problem eventually was the firewall. I didn't have the full co= ntrol over the server so I couldn't check the iptables rules.
The incoming traffic on 7000 port was rejected on the second node. I'v= e just checked one way communication from node2 -> node1 using telnet.
It was very helpful to run cassandra in TRACE mode. It displays a lot = of logs but =C2=A0was able to compare the=C2=A0behavior=C2=A0with a working= ring.
I've noticed that the second node was sending a Gossip= SYN message but was never getting back the SYN_ACK.

Anyway to everybody that will have a problem like that,= check your firewall settings:)

Thanks for help Roshni= .

--
regards,
Jakub Glapa


On Fri, Aug 3, 2012 at 9:41 AM, Roshni R= ajagopal <Roshni.Rajagopal@wal-mart.com> wrote:<= br>
Hi Jakub,

=C2=A0Were you able to resolve the issue?
For a multi data center setup I do believe some steps are different. You ma= y need to set Networktopology as your replication strategy rather than simp= le strategy, and setup a snitch. And mention the rack/dc configurations in = a config file.
You can refer to the steps for a multi data center installation.

Regards,
Roshni



From: Jakub Glapa <jakub.glapa@= gmail.com<mailto:jakub.glap= a@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:<= a href=3D"mailto:user@cassandra.apache.org">user@cassandra.apache.org&g= t;>
To: "user@cassandra.apach= e.org<mailto:user@cassa= ndra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Unsuccessful attempt to add a second node to a ring.

I found a similar thread from March : http://www.mail-archive.com/user@cassandra.apache.org/msg21007.= html

For me clearing the data and starting from the beginning didn't help.
It's interesting because on my dev environment I was able to add anothe= r node without any problems.

The only difference is that the second node now is in a different data cent= er. (but I'm not using any different settings, SimpleSnitch)
7000,9160,7199 ports were open between those 2 nodes.

How else can I check if the communication between those 2 nodes is working?=
In the logs I see that:
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 OutboundTcpConnection.= java (line 206) attempting to connect to NODE1/node1.ip

So I assume that the communication is somehow established?


--
regards,
Jakub Glapa


On Wed, Aug 1, 2012 at 11:36 AM, Jakub Glapa <jakub.glapa@gmail.com<mailto:= jakub.glapa@gmail.com>> = wrote:
yes it's the same



--
regards,
pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 11:24 AM, Roshni Rajagopal &= lt;Roshni.Rajagopal@wal-ma= rt.com<mailto:Roshn= i.Rajagopal@wal-mart.com>> wrote:
Ok, sorry it may not be required,
I was thinking of a configuration I had done on my local laptop, where I ha= d aliased my IP address.
In that case the directories and jmx port needed to be different.

Cluster name is same right?


From: Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>>
Reply-To: "user@cassandra= .apache.org<mailto:user= @cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" <user@cassandra.apache.org<mail= to:user@cassandra.apache.org><mailto:user@cassandr= a.apache.org<mailto:use= r@cassandra.apache.org>>>
To: "user@cassandra.apach= e.org<mailto:user@cassa= ndra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>= <mailto:user@cassandra.apac= he.org<mailto:user@cass= andra.apache.org>>>
Subject: Re: Unsuccessful attempt to add a sec= ond node to a ring.

Hi Roshni,
no they are the same, my changes in cassandra.yaml were only in the listen_= address, rpc_address, seeds and initial_token field.
The rest is exactly the same as on node1.

That's how the file looks on node2:



cluster_name: 'Test Cluster'
initial_token: 85070591730234615865843651857942052864
hinted_handoff_enabled: true
hinted_handoff_throttle_delay_in_ms: 1
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partitioner: org.apache.cassandra.dht.RandomPartitioner
data_file_directories:
=C2=A0 =C2=A0 - /data/servers/cassandra_sbe_edtool/cassandra_data/data
commitlog_directory: /data/servers/cassandra_sbe_edtool/cassandra_data/comm= itlog
saved_caches_directory: /data/servers/cassandra_sbe_edtool/cassandra_data/s= aved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
seed_provider:
=C2=A0 =C2=A0 - class_name: org.apache.cassandra.locator.SimpleSeedProvider=
=C2=A0 =C2=A0 =C2=A0 parameters:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - seeds: "NODE1"
flush_largest_memtables_at: 0.75
reduce_cache_sizes_at: 0.85
reduce_cache_capacity_to: 0.6
concurrent_reads: 32
concurrent_writes: 32
memtable_flush_queue_size: 4
sliced_buffer_size_in_kb: 64
storage_port: 7000
ssl_storage_port: 7001
listen_address: NODE2
rpc_address: NODE2
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: false
snapshot_before_compaction: false
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compaction: false
compaction_throughput_mb_per_sec: 16
compaction_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
index_interval: 128
encryption_options:
=C2=A0 =C2=A0 internode_encryption: none
=C2=A0 =C2=A0 keystore: conf/.keystore
=C2=A0 =C2=A0 keystore_password: cassandra
=C2=A0 =C2=A0 truststore: conf/.truststore
=C2=A0 =C2=A0 truststore_password: cassandra




--
regards,
pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajag= opal <Roshni.Rajagopal@= wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com><mailto:Roshni.Rajagopal@wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com= >>> wrote:
Jakub,

Have you set the
Data, commitlog, saved cache directories to different ones in each yaml fil= e for each node?

Regards,
Roshni


From: Jakub Glapa <jakub.= glapa@gmail.com<mailto:jaku= b.glapa@gmail.com><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>>>
Reply-To: "user@cassandra= .apache.org<mailto:user= @cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:user@cassandra.apache.org<mail= to:user@cassandra.apache.org><mailto:user@cassandr= a.apache.org<mailto:use= r@cassandra.apache.org>>>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<= ;mailto:user@cassandra.apache.= org>><mailto:user= @cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>>=
To: "user@cassandra.apach= e.org<mailto:user@cassa= ndra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>= <mailto:user@cassandra.apac= he.org<mailto:user@cass= andra.apache.org>>>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mail= to:user@cassandra.apache.org>><mailto:user@cass= andra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>>
Subject: Unsuccessful attempt to ad= d a second node to a ring.

Hi Everybody!

I'm trying to add a second node to an already operating one node cluste= r.

Some specs:
- cassandra 1.0.7
- both nodes have a routable listen_address and rpc_address.
- Ports are open: (from node2) telnet node1 7000 is successful
- Seeds parameter on node2 points to node 1.

[node1] nodetool -h localhost ring
Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 DC =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Ra= ck =C2=A0 =C2=A0 =C2=A0 =C2=A0Status State =C2=A0 Load =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token
node1.ip =C2=A0 =C2=A0 datacenter1 rack1 =C2=A0 =C2=A0 =C2=A0 Up =C2=A0 =C2= =A0 Normal =C2=A074.33 KB =C2=A0 =C2=A0 =C2=A0 =C2=A0100.00% 0

- initial token on node2 was specified

I see something like that in the logs on node2:

DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76) col= lectTimeOrderedData
=C2=A0INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667) JO= INING: waiting for ring and schema information
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 OutboundTcpConnection.= java (line 206) attempting to connect to NODE1/node1.ip
DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java (line= 86) Disseminating load info ...
=C2=A0INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667) JO= INING: schema complete, ready to bootstrap
DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ... got= ring + schema info
=C2=A0INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667) JO= INING: getting bootstrap token
DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token man= ually specified as 85070591730234615865843651857942052864
DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying mutatio= n of row 4c


but it doesn't join the ring:

[node2] nodetool -h localhost ring
Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 DC =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Ra= ck =C2=A0 =C2=A0 =C2=A0 =C2=A0Status State =C2=A0 Load =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token
node2.ip =C2=A0 datacenter1 rack1 =C2=A0 =C2=A0 =C2=A0 Up =C2=A0 =C2=A0 Nor= mal =C2=A013.49 KB =C2=A0 =C2=A0 =C2=A0 =C2=A0100.00% 850705917302346158658= 43651857942052864



I'm attaching the full log from node2 startup in debug mode.



PS.
When I didn't specified the initial token on node2 I ended up with exce= ption like that:
"Exception encountered during startup: No other nodes seen! =C2=A0Unab= le to bootstrap.If you intended to start a single-node cluster, you should = make sure your broadcast_address (or listen_address) is listed as a seed. Otherwise, you need to determine why the seed being contacted has no knowle= dge of the rest of the cluster. =C2=A0Usually, this can be solved by giving= all nodes the same seed list."


I'm not sure how to proceed now. I found a couple of posts with problem= s like that but they weren't very useful.

--
regards,
Jakub Glapa

This email and any files transmitted with it are confidential and intended = solely for the individual or entity to whom they are addressed. If you have= received this email in error destroy it immediately. *** Walmart Confident= ial ***

This email and any files transmitted with it are confidential and intended = solely for the individual or entity to whom they are addressed. If you have= received this email in error destroy it immediately. *** Walmart Confident= ial ***


This email and any files transmitted with it are confidential and intended = solely for the individual or entity to whom they are addressed. If you have= received this email in error destroy it immediately. *** Walmart Confident= ial ***

--f46d043c8024fbc40e04c65ca3f6--