cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Muth <doug.m...@gmail.com>
Subject Re: Nodetool talking to an old IP address (and timing out)
Date Tue, 29 May 2012 18:39:40 GMT
8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the
bottom of this.  Not having much of a Java or JMX background, I'll try
to explain it the best that I can.

To recap, my machine originally had the IP address of 10.244.207.16.
Then I shutdown/restarted that EC2 instance, and it had the IP
10.84.117.110.  During this, Cassandra was fine -- I could still
connect to 127.0.0.1 with cqlsh and the Helenus node.js module.

Things got weird only when I tried to use nodetool to manage the
instance.  As best I can tell, here's the algorithm that nodetool uses
when connecting to a Cassandra instance:

Step 1) Connect to the hostname and port specified on the command
line. "localhost" and "7199" are the defaults.

Step 2) Cassandra, at boot time, notes the hostname of the machine,
and tells nodetool "go connect to this hostname instead!"

After further investigation, it seems that after my instance was
rebooted, the file /etc/hostname was not updated.  It still had the
value "ip-10-244-207-16.ec2.internal" in it.  This means that any
attempt to connect to Cassandra involved Cassandra telling nodetool,
"Hey, go talk to 10.244.207.16 instead".  And that's where things went
wrong.

The permanent fix for this was to change the hostname to "localhost"
and to restart Cassandra.  The fact that Cassandra notes the hostname
at startup was one thing that made this so difficult to track down.  I
did not see the old IP anywhere in Cassandra configuration (or in
logfile output), so I did not think there was anything abnormal
happening in the instance.

While I'm sure there's a good reason for this sort of behavior, it is
very confusing to a Cassandra newbie such as myself, and I'll bet
others have been affected by this as well.  In the future, I think
some sort of logging of this sort of of logic, or perhaps a --verbose
mode for nodetool would be a really good idea.  What do other folks
think?

-- Doug
http://twitter.com/dmuth


On Tue, May 29, 2012 at 12:08 PM, Douglas Muth <doug.muth@gmail.com> wrote:
> I'm afraid that did not work.  I'm running JMX on port 7199 (the
> default) and I verified that the port is open and accepting
> connections.
[snip]

Mime
View raw message