cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuma BK" <>
Subject Re: Nodetool talking to an old IP address (and timing out)
Date Wed, 30 May 2012 05:03:16 GMT
Did you open the inbound port 1024 ~ 65535 at Security Group?

JMX uses two connection channels, one is 7199 by default for accepting 
connection request, another one is a random port between 1024 ~65535 decided 
during run time. Nodetool runs over JMX.


-----Original Message----- 
From: Douglas Muth
Sent: Tuesday, May 29, 2012 11:39 AM
Subject: Re: Nodetool talking to an old IP address (and timing out)

8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the
bottom of this.  Not having much of a Java or JMX background, I'll try
to explain it the best that I can.

To recap, my machine originally had the IP address of
Then I shutdown/restarted that EC2 instance, and it had the IP  During this, Cassandra was fine -- I could still
connect to with cqlsh and the Helenus node.js module.

Things got weird only when I tried to use nodetool to manage the
instance.  As best I can tell, here's the algorithm that nodetool uses
when connecting to a Cassandra instance:

Step 1) Connect to the hostname and port specified on the command
line. "localhost" and "7199" are the defaults.

Step 2) Cassandra, at boot time, notes the hostname of the machine,
and tells nodetool "go connect to this hostname instead!"

After further investigation, it seems that after my instance was
rebooted, the file /etc/hostname was not updated.  It still had the
value "ip-10-244-207-16.ec2.internal" in it.  This means that any
attempt to connect to Cassandra involved Cassandra telling nodetool,
"Hey, go talk to instead".  And that's where things went

The permanent fix for this was to change the hostname to "localhost"
and to restart Cassandra.  The fact that Cassandra notes the hostname
at startup was one thing that made this so difficult to track down.  I
did not see the old IP anywhere in Cassandra configuration (or in
logfile output), so I did not think there was anything abnormal
happening in the instance.

While I'm sure there's a good reason for this sort of behavior, it is
very confusing to a Cassandra newbie such as myself, and I'll bet
others have been affected by this as well.  In the future, I think
some sort of logging of this sort of of logic, or perhaps a --verbose
mode for nodetool would be a really good idea.  What do other folks

-- Doug

On Tue, May 29, 2012 at 12:08 PM, Douglas Muth <> wrote:
> I'm afraid that did not work.  I'm running JMX on port 7199 (the
> default) and I verified that the port is open and accepting
> connections.

View raw message