Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 36C93DB77 for ; Wed, 1 Aug 2012 09:52:30 +0000 (UTC) Received: (qmail 1524 invoked by uid 500); 1 Aug 2012 09:52:27 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 1332 invoked by uid 500); 1 Aug 2012 09:52:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 1284 invoked by uid 99); 1 Aug 2012 09:52:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Aug 2012 09:52:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jakub.glapa@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-ob0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Aug 2012 09:52:19 +0000 Received: by obbwc20 with SMTP id wc20so13679858obb.31 for ; Wed, 01 Aug 2012 02:51:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=zlvI3SOCmj/mARaGMk1HVCHuEbfCsQW0D4XyZDTuisQ=; b=BFJ6ClisbcNBd/PTtzJETRa722lt3Ke1j/GKTLOrAqpvLpmrFgODqFlcbxWjTymsln RHOardjfjiv3FqA2e8xosbH/z7ggcHyOZRONCNqJGPaVpcrUWSDQTFukz28JbFi4FBXv XmgsXDZU9LvhrMgN+zxWkjIbmpVljbYb/AvHgcKMw0NSxnn+of6Z+K7EEqnx2uwiwJNn MZbCYnFaeyEMCWzPkpXyjBdAVt5N/LRqVbsnwpfq5ercAv6AE1p81rWlBtkqadToTW01 f8hKgR5JSQlxFqYEhIzdcOE4G0UT2DjbnFt/Kyiko3b5M1G5F5u6f9CPRDydgR3+rXoN xGBg== MIME-Version: 1.0 Received: by 10.60.26.134 with SMTP id l6mr27788539oeg.40.1343814718118; Wed, 01 Aug 2012 02:51:58 -0700 (PDT) Received: by 10.182.50.234 with HTTP; Wed, 1 Aug 2012 02:51:58 -0700 (PDT) In-Reply-To: References: Date: Wed, 1 Aug 2012 10:51:58 +0100 Message-ID: Subject: Re: Unsuccessful attempt to add a second node to a ring. From: Jakub Glapa To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=e89a8ff1c63cead9ad04c6313ed7 --e89a8ff1c63cead9ad04c6313ed7 Content-Type: text/plain; charset=UTF-8 Hi Roshni, no they are the same, my changes in cassandra.yaml were only in the listen_address, rpc_address, seeds and initial_token field. The rest is exactly the same as on node1. That's how the file looks on node2: cluster_name: 'Test Cluster' initial_token: 85070591730234615865843651857942052864 hinted_handoff_enabled: true hinted_handoff_throttle_delay_in_ms: 1 authenticator: org.apache.cassandra.auth.AllowAllAuthenticator authority: org.apache.cassandra.auth.AllowAllAuthority partitioner: org.apache.cassandra.dht.RandomPartitioner data_file_directories: - /data/servers/cassandra_sbe_edtool/cassandra_data/data commitlog_directory: /data/servers/cassandra_sbe_edtool/cassandra_data/commitlog saved_caches_directory: /data/servers/cassandra_sbe_edtool/cassandra_data/saved_caches commitlog_sync: periodic commitlog_sync_period_in_ms: 10000 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "NODE1" flush_largest_memtables_at: 0.75 reduce_cache_sizes_at: 0.85 reduce_cache_capacity_to: 0.6 concurrent_reads: 32 concurrent_writes: 32 memtable_flush_queue_size: 4 sliced_buffer_size_in_kb: 64 storage_port: 7000 ssl_storage_port: 7001 listen_address: NODE2 rpc_address: NODE2 rpc_port: 9160 rpc_keepalive: true rpc_server_type: sync thrift_framed_transport_size_in_mb: 15 thrift_max_message_length_in_mb: 16 incremental_backups: false snapshot_before_compaction: false column_index_size_in_kb: 64 in_memory_compaction_limit_in_mb: 64 multithreaded_compaction: false compaction_throughput_mb_per_sec: 16 compaction_preheat_key_cache: true rpc_timeout_in_ms: 10000 endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch dynamic_snitch_update_interval_in_ms: 100 dynamic_snitch_reset_interval_in_ms: 600000 dynamic_snitch_badness_threshold: 0.1 request_scheduler: org.apache.cassandra.scheduler.NoScheduler index_interval: 128 encryption_options: internode_encryption: none keystore: conf/.keystore keystore_password: cassandra truststore: conf/.truststore truststore_password: cassandra -- regards, pozdrawiam, Jakub Glapa On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajagopal < Roshni.Rajagopal@wal-mart.com> wrote: > Jakub, > > Have you set the > Data, commitlog, saved cache directories to different ones in each yaml > file for each node? > > Regards, > Roshni > > > From: Jakub Glapa > > Reply-To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > Subject: Unsuccessful attempt to add a second node to a ring. > > Hi Everybody! > > I'm trying to add a second node to an already operating one node cluster. > > Some specs: > - cassandra 1.0.7 > - both nodes have a routable listen_address and rpc_address. > - Ports are open: (from node2) telnet node1 7000 is successful > - Seeds parameter on node2 points to node 1. > > [node1] nodetool -h localhost ring > Address DC Rack Status State Load > Owns Token > node1.ip datacenter1 rack1 Up Normal 74.33 KB > 100.00% 0 > > - initial token on node2 was specified > > I see something like that in the logs on node2: > > DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76) > collectTimeOrderedData > INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667) > JOINING: waiting for ring and schema information > DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 > OutboundTcpConnection.java (line 206) attempting to connect to > NODE1/node1.ip > DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java > (line 86) Disseminating load info ... > INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667) > JOINING: schema complete, ready to bootstrap > DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ... > got ring + schema info > INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667) > JOINING: getting bootstrap token > DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token > manually specified as 85070591730234615865843651857942052864 > DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying > mutation of row 4c > > > but it doesn't join the ring: > > [node2] nodetool -h localhost ring > Address DC Rack Status State Load > Owns Token > node2.ip datacenter1 rack1 Up Normal 13.49 KB 100.00% > 85070591730234615865843651857942052864 > > > > I'm attaching the full log from node2 startup in debug mode. > > > > PS. > When I didn't specified the initial token on node2 I ended up with > exception like that: > "Exception encountered during startup: No other nodes seen! Unable to > bootstrap.If you intended to start a single-node cluster, you should make > sure your broadcast_address (or listen_address) is listed as a seed. > Otherwise, you need to determine why the seed being contacted has no > knowledge of the rest of the cluster. Usually, this can be solved by > giving all nodes the same seed list." > > > I'm not sure how to proceed now. I found a couple of posts with problems > like that but they weren't very useful. > > -- > regards, > Jakub Glapa > > This email and any files transmitted with it are confidential and intended > solely for the individual or entity to whom they are addressed. If you have > received this email in error destroy it immediately. *** Walmart > Confidential *** > --e89a8ff1c63cead9ad04c6313ed7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Roshni,
no they are the same, my changes in cassandra.yaml were only= in the listen_address, rpc_address, seeds and initial_token field.=C2=A0
The rest is exactly the same as on node1.

Tha= t's how the file looks on node2:



cluster_name: 'T= est Cluster'
initial_token: 850705917302346158658436518579420= 52864
hinted_handoff_enabled: true
hinted_handoff_throt= tle_delay_in_ms: 1
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partiti= oner: org.apache.cassandra.dht.RandomPartitioner
data_file_direct= ories:
=C2=A0 =C2=A0 - /data/servers/cassandra_sbe_edtool/cassandra_data/data=
commitlog_directory: /data/servers/cassandra_sbe_edtool/cassandr= a_data/commitlog
saved_caches_directory: /data/servers/cassandra_= sbe_edtool/cassandra_data/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000<= /div>
seed_provider:
=C2=A0 =C2=A0 - class_name: org.apache.c= assandra.locator.SimpleSeedProvider
=C2=A0 =C2=A0 =C2=A0 paramete= rs:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - seeds: "NODE1"=
flush_largest_memtables_at: 0.75
reduce_cache_sizes_at: 0.85=
reduce_cache_capacity_to: 0.6
concurrent_reads: 32
concurrent_writes: 32
memtable_flush_queue_size: 4
sliced_buffer_size_in_kb: 64
storage_port: 7000
ss= l_storage_port: 7001
listen_address: NODE2
rpc_address:= NODE2
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: fa= lse
snapshot_before_compaction: false
column_index_size= _in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compactio= n: false
compaction_throughput_mb_per_sec: 16
compactio= n_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
dynami= c_snitch_update_interval_in_ms: 100=C2=A0
dynamic_snitch_reset_in= terval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
<= div> request_scheduler: org.apache.cassandra.scheduler.NoScheduler
ind= ex_interval: 128
encryption_options:
=C2=A0 =C2=A0 inte= rnode_encryption: none
=C2=A0 =C2=A0 keystore: conf/.keystore
=C2=A0 =C2=A0 keystore_password: cassandra
=C2=A0 =C2=A0 truststore: conf/.truststore
=C2=A0 =C2=A0 tru= ststore_password: cassandra




--
regards,<= br>pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 10:29 AM, Roshni = Rajagopal <Roshni.Rajagopal@wal-mart.com> wrote:=
Jakub,

Have you set the
Data, commitlog, saved cache directories to different ones in each yaml fil= e for each node?

Regards,
Roshni


From: Jakub Glapa <jakub.glapa@= gmail.com<mailto:jakub.glap= a@gmail.com>>
Reply-To: "user@cassandra= .apache.org<mailto:user= @cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
To: "user@cassandra.apach= e.org<mailto:user@cassa= ndra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Unsuccessful attempt to add a second node to a ring.

Hi Everybody!

I'm trying to add a second node to an already operating one node cluste= r.

Some specs:
- cassandra 1.0.7
- both nodes have a routable listen_address and rpc_address.
- Ports are open: (from node2) telnet node1 7000 is successful
- Seeds parameter on node2 points to node 1.

[node1] nodetool -h localhost ring
Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 DC =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Ra= ck =C2=A0 =C2=A0 =C2=A0 =C2=A0Status State =C2=A0 Load =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token
node1.ip =C2=A0 =C2=A0 datacenter1 rack1 =C2=A0 =C2=A0 =C2=A0 Up =C2=A0 =C2= =A0 Normal =C2=A074.33 KB =C2=A0 =C2=A0 =C2=A0 =C2=A0100.00% 0

- initial token on node2 was specified

I see something like that in the logs on node2:

DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76) col= lectTimeOrderedData
=C2=A0INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667) JO= INING: waiting for ring and schema information
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 OutboundTcpConnection.= java (line 206) attempting to connect to NODE1/node1.ip
DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java (line= 86) Disseminating load info ...
=C2=A0INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667) JO= INING: schema complete, ready to bootstrap
DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ... got= ring + schema info
=C2=A0INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667) JO= INING: getting bootstrap token
DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token man= ually specified as 85070591730234615865843651857942052864
DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying mutatio= n of row 4c


but it doesn't join the ring:

[node2] nodetool -h localhost ring
Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 DC =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Ra= ck =C2=A0 =C2=A0 =C2=A0 =C2=A0Status State =C2=A0 Load =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token
node2.ip =C2=A0 datacenter1 rack1 =C2=A0 =C2=A0 =C2=A0 Up =C2=A0 =C2=A0 Nor= mal =C2=A013.49 KB =C2=A0 =C2=A0 =C2=A0 =C2=A0100.00% 850705917302346158658= 43651857942052864



I'm attaching the full log from node2 startup in debug mode.



PS.
When I didn't specified the initial token on node2 I ended up with exce= ption like that:
"Exception encountered during startup: No other nodes seen! =C2=A0Unab= le to bootstrap.If you intended to start a single-node cluster, you should = make sure your broadcast_address (or listen_address) is listed as a seed. Otherwise, you need to determine why the seed being contacted has no knowle= dge of the rest of the cluster. =C2=A0Usually, this can be solved by giving= all nodes the same seed list."


I'm not sure how to proceed now. I found a couple of posts with problem= s like that but they weren't very useful.

--
regards,
Jakub Glapa

This email and any files transmitted with it are confidential a= nd intended solely for the individual or entity to whom they are addressed.= If you have received this email in error destroy it immediately. *** Walma= rt Confidential ***

--e89a8ff1c63cead9ad04c6313ed7--