Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3F15862B4 for ; Wed, 22 Jun 2011 16:47:45 +0000 (UTC) Received: (qmail 50057 invoked by uid 500); 22 Jun 2011 16:47:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 49982 invoked by uid 500); 22 Jun 2011 16:47:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 49974 invoked by uid 99); 22 Jun 2011 16:47:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 16:47:42 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nick@datastax.com designates 209.85.212.44 as permitted sender) Received: from [209.85.212.44] (HELO mail-vw0-f44.google.com) (209.85.212.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 16:47:35 +0000 Received: by vws12 with SMTP id 12so1011528vws.31 for ; Wed, 22 Jun 2011 09:47:14 -0700 (PDT) Received: by 10.52.177.70 with SMTP id co6mr1199332vdc.92.1308761234082; Wed, 22 Jun 2011 09:47:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.111.230 with HTTP; Wed, 22 Jun 2011 09:46:54 -0700 (PDT) X-Originating-IP: [64.132.24.216] In-Reply-To: <797C8A77-5F5C-43C6-BC27-CD7975B331E4@gmail.com> References: <797C8A77-5F5C-43C6-BC27-CD7975B331E4@gmail.com> From: Nick Bailey Date: Wed, 22 Jun 2011 11:46:54 -0500 Message-ID: Subject: Re: Strange Connection error of nodetool To: user@cassandra.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This is almost certainly caused by the weird connection process JMX uses. JMX actually uses a 2 connection process, the second connection is determined by the 'JVM_OPTS=3D"$JVM_OPTS -Djava.rmi.server.hostname=3D"' setting in your cassandra-env.sh configuration file. By default that setting is commented out and the hostname will default to whatever the local machine thinks it's hostname is. You should be able to fix this by uncommenting that setting in cassandra-env.sh and using the correct hostname for that setting. On Wed, Jun 22, 2011 at 4:02 AM, =EB=B0=95=EC=83=81=EA=B8=B8 wrote: > Hi. > > I'm running 5 cassandra nodes. Say, the addresses are 112.234.123.111 ~ 1= 12.234.123.115; the real address is different. > When I run nodetool, the one node of address 112.234.123.112 has failed t= o connect. Showing error message like this. > > iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring > Error connection to remote JMX agent! > java.rmi.ConnectException: Connection refused to host: 122.234.123.112; n= ested exception is: > > The host to connect address differ! I had tried to query 112.* but, the n= odetool tried to connect 122.*. It happened just one machine. All other mac= hines works fine. > And I can connect to 112.234.123.112 by cassandra-cli or other tools usin= g other port (such as 22 of ssh, 80 of http). It has trouble only on nodeto= ol. > > Does anyone has an idea? > > I'll paste the full stack trace below. > > iPark:~ hayarobi$ nodetool --host 112.234.123.111 ring > Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 Status State =C2=A0 Load =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 136112946768375000000000000000000= 000000 > 112.234.123.111 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0725.01 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A00 > 112.234.123.112 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0725.93 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A034028236692093800000000000000000000000 > 112.234.123.113 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0728.2 KB =C2=A0 =C2= =A0 =C2=A0 =C2=A020.00% =C2=A068056473384187700000000000000000000000 > 112.234.123.114 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0713.1 KB =C2=A0 =C2= =A0 =C2=A0 =C2=A020.00% =C2=A0102084710076282000000000000000000000000 > 112.234.123.115 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0722.67 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A0136112946768375000000000000000000000000 > iPark:~ hayarobi$ nodetool --host 112.234.123.115 ring > Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 Status State =C2=A0 Load =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 136112946768375000000000000000000= 000000 > 112.234.123.111 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0725.01 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A00 > 112.234.123.112 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0725.93 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A034028236692093800000000000000000000000 > 112.234.123.113 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0728.2 KB =C2=A0 =C2= =A0 =C2=A0 =C2=A020.00% =C2=A068056473384187700000000000000000000000 > 112.234.123.114 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0713.1 KB =C2=A0 =C2= =A0 =C2=A0 =C2=A020.00% =C2=A0102084710076282000000000000000000000000 > 112.234.123.115 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0722.67 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A0136112946768375000000000000000000000000 > iPark:~ hayarobi$ nodetool --host 112.234.123.114 ring > Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 Status State =C2=A0 Load =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Owns =C2=A0 =C2=A0Token > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 136112946768375000000000000000000= 000000 > 112.234.123.111 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0725.01 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A00 > 112.234.123.112 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0725.93 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A034028236692093800000000000000000000000 > 112.234.123.113 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0728.2 KB =C2=A0 =C2= =A0 =C2=A0 =C2=A020.00% =C2=A068056473384187700000000000000000000000 > 112.234.123.114 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0713.1 KB =C2=A0 =C2= =A0 =C2=A0 =C2=A020.00% =C2=A0102084710076282000000000000000000000000 > 112.234.123.115 =C2=A0Up =C2=A0 =C2=A0 Normal =C2=A0722.67 KB =C2=A0 =C2= =A0 =C2=A0 20.00% =C2=A0136112946768375000000000000000000000000 > iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring > Error connection to remote JMX agent! > java.rmi.ConnectException: Connection refused to host: 122.234.123.112; n= ested exception is: > =C2=A0 =C2=A0 =C2=A0 =C2=A0java.net.ConnectException: Connection refused > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.transport.tcp.TCPEndpoint.newSocket= (TCPEndpoint.java:601) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.transport.tcp.TCPChannel.createConn= ection(TCPChannel.java:198) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.transport.tcp.TCPChannel.newConnect= ion(TCPChannel.java:184) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.server.UnicastRef.invoke(UnicastRef= .java:110) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.management.remote.rmi.RMIServerImpl_S= tub.newClient(Unknown Source) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.management.remote.rmi.RMIConnector.ge= tConnection(RMIConnector.java:2327) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.management.remote.rmi.RMIConnector.co= nnect(RMIConnector.java:279) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.management.remote.JMXConnectorFactory= .connect(JMXConnectorFactory.java:248) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.tools.NodeProbe.connec= t(NodeProbe.java:137) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.tools.NodeProbe.= (NodeProbe.java:107) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.tools.NodeCmd.main(Nod= eCmd.java:511) > Caused by: java.net.ConnectException: Connection refused > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.PlainSocketImpl.socketConnect(Nati= ve Method) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.PlainSocketImpl.doConnect(PlainSoc= ketImpl.java:351) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.PlainSocketImpl.connectToAddress(P= lainSocketImpl.java:213) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.PlainSocketImpl.connect(PlainSocke= tImpl.java:200) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.SocksSocketImpl.connect(SocksSocke= tImpl.java:432) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.Socket.connect(Socket.java:529) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.Socket.connect(Socket.java:478) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.Socket.(Socket.java:375) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.Socket.(Socket.java:189) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.transport.proxy.RMIDirectSocketFact= ory.createSocket(RMIDirectSocketFactory.java:22) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.transport.proxy.RMIMasterSocketFact= ory.createSocket(RMIMasterSocketFactory.java:128) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.rmi.transport.tcp.TCPEndpoint.newSocket= (TCPEndpoint.java:595) > =C2=A0 =C2=A0 =C2=A0 =C2=A0... 10 more > >