Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9BC2664F2 for ; Wed, 22 Jun 2011 23:16:11 +0000 (UTC) Received: (qmail 81033 invoked by uid 500); 22 Jun 2011 23:16:09 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81002 invoked by uid 500); 22 Jun 2011 23:16:09 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 80994 invoked by uid 99); 22 Jun 2011 23:16:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:16:08 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a49.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:16:02 +0000 Received: from homiemail-a49.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a49.g.dreamhost.com (Postfix) with ESMTP id 2CA0B5E006B for ; Wed, 22 Jun 2011 16:15:42 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=15tYHF0ihC qSo8NODOfpqIIiMZ9etXAoaxhhU4P4A2hk+zVGL6xIbiKT+N6mpHZuyYiKldH3Yo 927eNL5EGSYmTQ4f/P110Ae5Po0EqNXCvsPFk6264nq6bV5SGiQRhR8bIAQxUFmI S0uDv62i/1bgR3ZHrLoyQM7ADTcS4g274= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=HyrcycXON4jFRmQE CzJhml0bSak=; b=nVmAml6vljjQf2reS3jwlz6HXDGYivfvSorwRNuNIBXB+X7W FtH1cbJUNF4wBgumIfTrWXudmaPYVV9FrKns/WIQauYYmOQ3M4yKZB7uMzI7lKAj OQfyiWZ2FDp8HWrhd7imbZg2suMriEcxedxKBmpWejrEW5RcbtPzYq0SvU4= Received: from [10.0.1.151] (121-73-157-230.cable.telstraclear.net [121.73.157.230]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a49.g.dreamhost.com (Postfix) with ESMTPSA id 61F615E0058 for ; Wed, 22 Jun 2011 16:15:41 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-10--235142876 Subject: Re: Strange Connection error of nodetool Date: Thu, 23 Jun 2011 11:15:38 +1200 In-Reply-To: <797C8A77-5F5C-43C6-BC27-CD7975B331E4@gmail.com> To: user@cassandra.apache.org References: <797C8A77-5F5C-43C6-BC27-CD7975B331E4@gmail.com> Message-Id: X-Mailer: Apple Mail (2.1084) --Apple-Mail-10--235142876 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Check the list here=20 http://wiki.apache.org/cassandra/JmxGotchas I *think* the jmx server tells the client to connect back on another = host/port. Hope that helps.=20 =20 ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 22 Jun 2011, at 21:02, =EB=B0=95=EC=83=81=EA=B8=B8 wrote: > Hi. >=20 > I'm running 5 cassandra nodes. Say, the addresses are 112.234.123.111 = ~ 112.234.123.115; the real address is different.=20 > When I run nodetool, the one node of address 112.234.123.112 has = failed to connect. Showing error message like this.=20 >=20 > iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring > Error connection to remote JMX agent! > java.rmi.ConnectException: Connection refused to host: = 122.234.123.112; nested exception is:=20 >=20 > The host to connect address differ! I had tried to query 112.* but, = the nodetool tried to connect 122.*. It happened just one machine. All = other machines works fine.=20 > And I can connect to 112.234.123.112 by cassandra-cli or other tools = using other port (such as 22 of ssh, 80 of http). It has trouble only on = nodetool. >=20 > Does anyone has an idea?=20 >=20 > I'll paste the full stack trace below. >=20 > iPark:~ hayarobi$ nodetool --host 112.234.123.111 ring > Address Status State Load Owns Token = =20 > = 136112946768375000000000000000000000000 =20 > 112.234.123.111 Up Normal 725.01 KB 20.00% 0 = =20 > 112.234.123.112 Up Normal 725.93 KB 20.00% = 34028236692093800000000000000000000000 =20 > 112.234.123.113 Up Normal 728.2 KB 20.00% = 68056473384187700000000000000000000000 =20 > 112.234.123.114 Up Normal 713.1 KB 20.00% = 102084710076282000000000000000000000000 =20 > 112.234.123.115 Up Normal 722.67 KB 20.00% = 136112946768375000000000000000000000000 =20 > iPark:~ hayarobi$ nodetool --host 112.234.123.115 ring > Address Status State Load Owns Token = =20 > = 136112946768375000000000000000000000000 =20 > 112.234.123.111 Up Normal 725.01 KB 20.00% 0 = =20 > 112.234.123.112 Up Normal 725.93 KB 20.00% = 34028236692093800000000000000000000000 =20 > 112.234.123.113 Up Normal 728.2 KB 20.00% = 68056473384187700000000000000000000000 =20 > 112.234.123.114 Up Normal 713.1 KB 20.00% = 102084710076282000000000000000000000000 =20 > 112.234.123.115 Up Normal 722.67 KB 20.00% = 136112946768375000000000000000000000000 =20 > iPark:~ hayarobi$ nodetool --host 112.234.123.114 ring > Address Status State Load Owns Token = =20 > = 136112946768375000000000000000000000000 =20 > 112.234.123.111 Up Normal 725.01 KB 20.00% 0 = =20 > 112.234.123.112 Up Normal 725.93 KB 20.00% = 34028236692093800000000000000000000000 =20 > 112.234.123.113 Up Normal 728.2 KB 20.00% = 68056473384187700000000000000000000000 =20 > 112.234.123.114 Up Normal 713.1 KB 20.00% = 102084710076282000000000000000000000000 =20 > 112.234.123.115 Up Normal 722.67 KB 20.00% = 136112946768375000000000000000000000000 =20 > iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring > Error connection to remote JMX agent! > java.rmi.ConnectException: Connection refused to host: = 122.234.123.112; nested exception is:=20 > java.net.ConnectException: Connection refused > at = sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) > at = sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) > at = sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110) > at = javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) > at = javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2= 327) > at = javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279) > at = javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.ja= va:248) > at = org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137) > at = org.apache.cassandra.tools.NodeProbe.(NodeProbe.java:107) > at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511) > Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > at = java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432) > at java.net.Socket.connect(Socket.java:529) > at java.net.Socket.connect(Socket.java:478) > at java.net.Socket.(Socket.java:375) > at java.net.Socket.(Socket.java:189) > at = sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocke= tFactory.java:22) > at = sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocke= tFactory.java:128) > at = sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) > ... 10 more >=20 --Apple-Mail-10--235142876 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Check = the list here 

I *think* the = jmx server tells the client to connect back on another = host/port.

Hope that = helps. 
 
http://www.thelastpickle.com

On 22 Jun 2011, at 21:02, =EB=B0=95=EC=83=81=EA=B8=B8 = wrote:

Hi.

I'm running 5 cassandra nodes. Say, the = addresses are 112.234.123.111 ~ 112.234.123.115; the real address is = different.
When I run nodetool, the one node of address = 112.234.123.112 has failed to connect. Showing error message like this. =

iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
Error = connection to remote JMX agent!
java.rmi.ConnectException: Connection = refused to host: 122.234.123.112; nested exception is:

The host = to connect address differ! I had tried to query 112.* but, the nodetool = tried to connect 122.*. It happened just one machine. All other machines = works fine.
And I can connect to 112.234.123.112 by cassandra-cli or = other tools using other port (such as 22 of ssh, 80 of http). It has = trouble only on nodetool.

Does anyone has an idea?

I'll = paste the full stack trace below.

iPark:~ hayarobi$ nodetool = --host 112.234.123.111 ring
Address =         Status State =   Load =            Owns =    Token =             &n= bsp;           &nbs= p;            =  
=             &n= bsp;           &nbs= p;            =             &n= bsp;    136112946768375000000000000000000000000 =     
112.234.123.111  Up =     Normal  725.01 KB =       20.00%  0 =             &n= bsp;           &nbs= p;            =      
112.234.123.112  Up =     Normal  725.93 KB =       20.00% =  34028236692093800000000000000000000000 =      
112.234.123.113  Up =     Normal  728.2 KB =        20.00% =  68056473384187700000000000000000000000 =      
112.234.123.114  Up =     Normal  713.1 KB =        20.00% =  102084710076282000000000000000000000000 =     
112.234.123.115  Up =     Normal  722.67 KB =       20.00% =  136112946768375000000000000000000000000 =     
iPark:~ hayarobi$ nodetool --host = 112.234.123.115 ring
Address =         Status State =   Load =            Owns =    Token =             &n= bsp;           &nbs= p;            =  
=             &n= bsp;           &nbs= p;            =             &n= bsp;    136112946768375000000000000000000000000 =     
112.234.123.111  Up =     Normal  725.01 KB =       20.00%  0 =             &n= bsp;           &nbs= p;            =      
112.234.123.112  Up =     Normal  725.93 KB =       20.00% =  34028236692093800000000000000000000000 =      
112.234.123.113  Up =     Normal  728.2 KB =        20.00% =  68056473384187700000000000000000000000 =      
112.234.123.114  Up =     Normal  713.1 KB =        20.00% =  102084710076282000000000000000000000000 =     
112.234.123.115  Up =     Normal  722.67 KB =       20.00% =  136112946768375000000000000000000000000 =     
iPark:~ hayarobi$ nodetool --host = 112.234.123.114 ring
Address =         Status State =   Load =            Owns =    Token =             &n= bsp;           &nbs= p;            =  
=             &n= bsp;           &nbs= p;            =             &n= bsp;    136112946768375000000000000000000000000 =     
112.234.123.111  Up =     Normal  725.01 KB =       20.00%  0 =             &n= bsp;           &nbs= p;            =      
112.234.123.112  Up =     Normal  725.93 KB =       20.00% =  34028236692093800000000000000000000000 =      
112.234.123.113  Up =     Normal  728.2 KB =        20.00% =  68056473384187700000000000000000000000 =      
112.234.123.114  Up =     Normal  713.1 KB =        20.00% =  102084710076282000000000000000000000000 =     
112.234.123.115  Up =     Normal  722.67 KB =       20.00% =  136112946768375000000000000000000000000 =     
iPark:~ hayarobi$ nodetool --host = 112.234.123.112 ring
Error connection to remote JMX = agent!
java.rmi.ConnectException: Connection refused to host: = 122.234.123.112; nested exception is:
=        java.net.ConnectException: = Connection refused
       at = sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
=        at = sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
=        at = sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
=        at = sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
=        at = javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown = Source)
       at = javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2= 327)
       at = javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)        at = javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.ja= va:248)
       at = org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
=        at = org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
=        at = org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
Caused by: = java.net.ConnectException: Connection refused
=        at = java.net.PlainSocketImpl.socketConnect(Native Method)
=        at = java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
=        at = java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
=        at = java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
=        at = java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
=        at = java.net.Socket.connect(Socket.java:529)
=        at = java.net.Socket.connect(Socket.java:478)
=        at = java.net.Socket.<init>(Socket.java:375)
=        at = java.net.Socket.<init>(Socket.java:189)
=        at = sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocke= tFactory.java:22)
       at = sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocke= tFactory.java:128)
       at = sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
=        ... 10 = more


= --Apple-Mail-10--235142876--