Hi Roshni, 
I was just about to summarize. I got it running.
The problem eventually was the firewall. I didn't have the full control over the server so I couldn't check the iptables rules.
The incoming traffic on 7000 port was rejected on the second node. I've just checked one way communication from node2 -> node1 using telnet.
It was very helpful to run cassandra in TRACE mode. It displays a lot of logs but  was able to compare the behavior with a working ring.
I've noticed that the second node was sending a Gossip SYN message but was never getting back the SYN_ACK.

Anyway to everybody that will have a problem like that, check your firewall settings:)

Thanks for help Roshni.

--
regards,
Jakub Glapa


On Fri, Aug 3, 2012 at 9:41 AM, Roshni Rajagopal <Roshni.Rajagopal@wal-mart.com> wrote:
Hi Jakub,

 Were you able to resolve the issue?
For a multi data center setup I do believe some steps are different. You may need to set Networktopology as your replication strategy rather than simple strategy, and setup a snitch. And mention the rack/dc configurations in a config file.
You can refer to the steps for a multi data center installation.

Regards,
Roshni



From: Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>
I found a similar thread from March : http://www.mail-archive.com/user@cassandra.apache.org/msg21007.html

For me clearing the data and starting from the beginning didn't help.

It's interesting because on my dev environment I was able to add another node without any problems.

The only difference is that the second node now is in a different data center. (but I'm not using any different settings, SimpleSnitch)
7000,9160,7199 ports were open between those 2 nodes.

How else can I check if the communication between those 2 nodes is working?
In the logs I see that:
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 OutboundTcpConnection.java (line 206) attempting to connect to NODE1/node1.ip

So I assume that the communication is somehow established?


--
regards,
Jakub Glapa


On Wed, Aug 1, 2012 at 11:36 AM, Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>> wrote:
yes it's the same



--
regards,
pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 11:24 AM, Roshni Rajagopal <Roshni.Rajagopal@wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com>> wrote:
Ok, sorry it may not be required,
I was thinking of a configuration I had done on my local laptop, where I had aliased my IP address.
In that case the directories and jmx port needed to be different.

Cluster name is same right?


Subject: Re: Unsuccessful attempt to add a second node to a ring.

Hi Roshni,
no they are the same, my changes in cassandra.yaml were only in the listen_address, rpc_address, seeds and initial_token field.
The rest is exactly the same as on node1.

That's how the file looks on node2:



cluster_name: 'Test Cluster'
initial_token: 85070591730234615865843651857942052864
hinted_handoff_enabled: true
hinted_handoff_throttle_delay_in_ms: 1
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partitioner: org.apache.cassandra.dht.RandomPartitioner
data_file_directories:
    - /data/servers/cassandra_sbe_edtool/cassandra_data/data
commitlog_directory: /data/servers/cassandra_sbe_edtool/cassandra_data/commitlog
saved_caches_directory: /data/servers/cassandra_sbe_edtool/cassandra_data/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "NODE1"
flush_largest_memtables_at: 0.75
reduce_cache_sizes_at: 0.85
reduce_cache_capacity_to: 0.6
concurrent_reads: 32
concurrent_writes: 32
memtable_flush_queue_size: 4
sliced_buffer_size_in_kb: 64
storage_port: 7000
ssl_storage_port: 7001
listen_address: NODE2
rpc_address: NODE2
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: false
snapshot_before_compaction: false
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compaction: false
compaction_throughput_mb_per_sec: 16
compaction_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
index_interval: 128
encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra




--
regards,
pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajagopal <Roshni.Rajagopal@wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com><mailto:Roshni.Rajagopal@wal-mart.com<mailto:Roshni.Rajagopal@wal-mart.com>>> wrote:
Jakub,

Have you set the
Data, commitlog, saved cache directories to different ones in each yaml file for each node?

Regards,
Roshni


From: Jakub Glapa <jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com><mailto:jakub.glapa@gmail.com<mailto:jakub.glapa@gmail.com>>>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>>
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>>
Subject: Unsuccessful attempt to add a second node to a ring.

Hi Everybody!

I'm trying to add a second node to an already operating one node cluster.

Some specs:
- cassandra 1.0.7
- both nodes have a routable listen_address and rpc_address.
- Ports are open: (from node2) telnet node1 7000 is successful
- Seeds parameter on node2 points to node 1.

[node1] nodetool -h localhost ring
Address         DC          Rack        Status State   Load            Owns    Token
node1.ip     datacenter1 rack1       Up     Normal  74.33 KB        100.00% 0

- initial token on node2 was specified

I see something like that in the logs on node2:

DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76) collectTimeOrderedData
 INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667) JOINING: waiting for ring and schema information
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642 OutboundTcpConnection.java (line 206) attempting to connect to NODE1/node1.ip
DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java (line 86) Disseminating load info ...
 INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667) JOINING: schema complete, ready to bootstrap
DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ... got ring + schema info
 INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667) JOINING: getting bootstrap token
DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token manually specified as 85070591730234615865843651857942052864
DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying mutation of row 4c


but it doesn't join the ring:

[node2] nodetool -h localhost ring
Address         DC          Rack        Status State   Load            Owns    Token
node2.ip   datacenter1 rack1       Up     Normal  13.49 KB        100.00% 85070591730234615865843651857942052864



I'm attaching the full log from node2 startup in debug mode.



PS.
When I didn't specified the initial token on node2 I ended up with exception like that:
"Exception encountered during startup: No other nodes seen!  Unable to bootstrap.If you intended to start a single-node cluster, you should make sure your broadcast_address (or listen_address) is listed as a seed.
Otherwise, you need to determine why the seed being contacted has no knowledge of the rest of the cluster.  Usually, this can be solved by giving all nodes the same seed list."


I'm not sure how to proceed now. I found a couple of posts with problems like that but they weren't very useful.

--
regards,
Jakub Glapa

This email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error destroy it immediately. *** Walmart Confidential ***

This email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error destroy it immediately. *** Walmart Confidential ***


This email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error destroy it immediately. *** Walmart Confidential ***