Thanks! That did it. Looks like the connection happens on 10036 and then the server negotiates a separate port for continued communication. 

Found this article once I knew what to look for. It also describes how to get more consistency on port numbers to allow for ssh tunneling and firewalls.

From http://jared.ottleys.net/alfresco/tunneling-debug-and-jmx-for-alfresco

"The -Djava.rmi.server.hostname=dummyhost option is needed to help RMI know where to connect.  RMI connects in a two part process.  First by connecting to the RMI server registry, which pushes your request to the JMX service which is dynamically allocated on the first open port available to it at start up time."


On Aug 30, 2010, at 10:30 AM, Juho Mäkinen wrote:

I think that JMX needs additional ports to function correctly. Try to
disable all firewalls between the client and the server so that client
can connect to any port in the server and try again.

- Juho Mäkinen

On Mon, Aug 30, 2010 at 7:07 PM, Allan Carroll <allanca@gmail.com> wrote:
Hi,

I'm trying to manage my cassandra cluster from a remote box and having issues getting nodetool to connect. All the machines I'm using are running on AWS.

Here's what happens when I try:

/opt/apache-cassandra-0.6.4/bin/nodetool -h xxx.xxx.xxx.143 -p 10036 ring
Error connecting to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: xxx.xxx.xxx.143; nested exception is:
       java.net.ConnectException: Connection timed out


When I'm local to a box (Ubuntu 10.04) running Cassandra, I can connect fine via both 127.0.0.1 and external ip (xxx.xxx.xxx.143). I can telnet into the jmx port from an external machine fine:

telnet xxx.xxx.xxx.143 10036
Trying xxx.xxx.xxx.143...
Connected to xxx.xxx.xxx.143.
Escape character is '^]'.

I already added the -Djava.rmi.server.hostname parameter to the java runtime, but it didn't seem to affect anything.

/usr/bin/jsvc -home /usr/lib/jvm/java-6-openjdk/jre -pidfile /var/run/cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/antlr-3.1.3.jar:/usr/share/cassandra/apache-cassandra-0.6.3.jar:/usr/share/cassandra/avro-1.2.0-dev.jar:/usr/share/cassandra/clhm-production.jar:/usr/share/cassandra/commons-cli-1.1.jar:/usr/share/cassandra/commons-codec-1.2.jar:/usr/share/cassandra/commons-collections-3.2.1.jar:/usr/share/cassandra/commons-lang-2.4.jar:/usr/share/cassandra/google-collections-1.0.jar:/usr/share/cassandra/hadoop-core-0.20.1.jar:/usr/share/cassandra/high-scale-lib.jar:/usr/share/cassandra/ivy-2.1.0.jar:/usr/share/cassandra/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/jline-0.9.94.jar:/usr/share/cassandra/json-simple-1.1.jar:/usr/share/cassandra/libthrift-r917130.jar:/usr/share/cassandra/log4j-1.2.14.jar:/usr/share/cassandra/slf4j-api-1.5.8.jar:/usr/share/cassandra/slf4j-log4j12-1.5.8.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Xmx4G -Xms128M -Djava.rmi.server.hostname=xxx.xxx.xxx.143 -Dcassandra -Dstorage-config=/etc/cassandra -Dcom.sun.management.jmxremote.port=10036 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon


netstat shows that I'm still bound to IP6

netstat -nap|grep 10036
tcp6       0      0 :::10036                :::*                    LISTEN      29277/jsvc


And, now I'm at an impasse. Any help would be greatly appreciated.

Thanks
-Allan