Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2016D17D1B for ; Tue, 24 Feb 2015 23:56:17 +0000 (UTC) Received: (qmail 72082 invoked by uid 500); 24 Feb 2015 23:56:13 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 72030 invoked by uid 500); 24 Feb 2015 23:56:13 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 72018 invoked by uid 500); 24 Feb 2015 23:56:13 -0000 Delivered-To: apmail-hadoop-zookeeper-user@hadoop.apache.org Received: (qmail 72015 invoked by uid 99); 24 Feb 2015 23:56:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2015 23:56:13 +0000 X-ASF-Spam-Status: No, hits=4.5 required=5.0 tests=FORGED_YAHOO_RCVD,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of fpjunqueira@yahoo.com designates 72.30.239.142 as permitted sender) Received: from [72.30.239.142] (HELO nm32-vm6.bullet.mail.bf1.yahoo.com) (72.30.239.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2015 23:56:08 +0000 Received: from [66.196.81.170] by nm32.bullet.mail.bf1.yahoo.com with NNFMP; 24 Feb 2015 23:52:38 -0000 Received: from [98.139.211.197] by tm16.bullet.mail.bf1.yahoo.com with NNFMP; 24 Feb 2015 23:52:38 -0000 Received: from [127.0.0.1] by smtp206.mail.bf1.yahoo.com with NNFMP; 24 Feb 2015 23:52:38 -0000 X-Yahoo-Newman-Id: 177747.56174.bm@smtp206.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: NcbroKQVM1nmxL.rKQ0FQhMn3jI0.Ws3R3zv_4Rwqb.A7sy Y32fhjNPnuwpVHrUqkrOR8M68YHwXBY7XJq_ZCUJwrHgkKDUzpmz3olYnx2c qiU_Pescv6hJeQlYA58uLSQKtu3p60.2xDR9yH4hWS1aHlxFVRioS1Rr_Pr9 BsnwGSVl_xNRw40G8r9XG2ReeE9ZxHXtzxpvol3dO1pYoolSpa.5DKlEOstH TAWQbhl44OY_qYozwa8yhLEBIYGC.ZzCaUaZR78x7Qmc91wGB3KII1bofOdp bxYceqoYQJvF0OrzrNwpZV47zdaFy.r8vq3Db278FxehrVi6Sc2SRwm18rPE jwNcM3boyB2vHVhdKmI5qLI6DYrfANSzS3_CLG5USwdAClbOuB0pcQNuaZew TqV7U.JXvL25ntJGh_RUVrjvcSLQlMlK1g56RfosxuLgtlyeMWUa57iCQlWn uPQDwJQrfmY5PDdWK5W7oOSdTt.2eHQG70t2wa_7XfV2YyttLjHQgcUVdhnk uX7DhsmE7xfadfl9kxWJbjocN4GWjwfAfAkhKTH.z5Csfz_6puXnh1N5vJC1 vJt3zPCrl115q6J0S9pN8iW3A_YiAgWRcqoPO3R3zcokqfKkZpfmLKVeM0xs CWeSQw03iueeigQ.Bji1SgANhiAIRLtDuxS04hVCK0BIk.pttMJu1rZP2JgL 6x95hmGUOo6oC7zZhJ1BngsrsDZHFUXfubO9mKyKb3fmVvLdis9neDXp1SSU ISXr5KXChM9iqI6_5wmF3qPMQurcuO2hvkiUQBKRzLZROpef_idczgBmF5vB QT4sWD2hjnmR8191d3aQzu15W_zZc_3Hd8sBZRrYTULE9YLOlPDqS6xxjOtF TH3Y3vazzwYQ6PUiR3CfJxJiZtSMB8lj2yRn9GH0PxIe79Ir8Zrwl_BisVx0 R1vVpx1f1vyJTRSfFGUeb10kinprDhPMH.Cfkiha4x1W7UxlsvLtUMzaTNEG mtkdbpt5sOgLQHEQfWhtQEiceojBIjLnJOQIcdh7XSXAw.yTti4w35SX5bNc 1sSpbIMSVweyrqRflBxoKg9rxwO3M22AnEM_a3C2wKyDNqdiLDjU.5GwH.NK yf.2Hvq25NZfo2E8uWRN1gGApzQHevpybeOeNT5sTG0LtjhvNFJVQtELG7CH Q_ccLlNviYzyRqOiylOjAQiTduBGVL_vsAYXbD9t5DwRMTRsgr1g8lr.9opP ZDm_75JzQwLq1nOSAkNqXJzeG2kOpbv7MyxsBv6CHzOvelgM3XEjfY8zmuaf YlVaxhXSX4_rfo.3wKNz9PSjrVI1d8tpJ1cRtmE_XMGs.7fm7IzObQulZphi toQucgh81E3TvLHLhxs_8VvhZvcRelD9GdkVnLQmObSXHEFZnySQ98LT4xGJ Yo.ASvM0Q1azkbxXKxx1G2rmzu0Ck4gkDpM6i7VGDEyy.5feQg92uAlePyZt d.LBYplMWWZCCmvYOmFxw4h2sUGBeJCu7XxOvQA.GWCjY_oGzWBuY6Fa0Qqi 9RTLz_LKYv40LUEfHkkG.i.sQNeFyW4Ztlv85CCYgfTevIRlD8wjb8C.iqej ajvekZiW2proiXg09eKPB0GJGu3iCb_45WL6.F5U9xIK_sqrm9laKrhTR.tS 2H9zIlMS0EDQWKzjHHdO_NpmxqHOJn6.kjbuSZYFTdrk5M9PIpVHHegdbcd7 LI59rvsoWo5z150WvhKyXc_uMmfj90j72PTKCeeXKJGMFRTDJwQtWmKMj9d. utTjNQ7cS.2D_gB9B3U8FPsgDKDj_9G_YHBMfH7ZOASrySLiP1ET98_qo9HD _kPrXMhTO1nszcit3o7xMVZUykLwXWuCm9XS8L8IaKDA7hl7EsusxM8CF6ZA RoY_DKn8uo2omCIAuYs5t8vViBt2gUEix5uTmTbKWZkDEW7mv2wrsu6aWn_1 zIr2X6pb1Lc3IxEYARQ-- X-Yahoo-SMTP: HT5UJDeswBACWJPOeualxAa.da..S.fl Content-Type: multipart/alternative; boundary="Apple-Mail=_C167C6B8-0357-4121-A2AB-58F548A2724D" Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: New Zookeeper Installation - connection refused From: Flavio Junqueira In-Reply-To: <443BCE38E921434394B45FF0813FA57850333322@MSGEXSIL1119.ent.wfb.bank.corp> Date: Tue, 24 Feb 2015 23:52:37 +0000 Cc: zookeeper-user@hadoop.apache.org Message-Id: References: <1084398278.2012304.1423041966808.JavaMail.yahoo@mail.yahoo.com> <443BCE38E921434394B45FF0813FA5785031021F@MSGEXSIL1119.ent.wfb.bank.corp> <1883587906.2086325.1423056777334.JavaMail.yahoo@mail.yahoo.com> <443BCE38E921434394B45FF0813FA57850310392@MSGEXSIL1119.ent.wfb.bank.corp> <54D23E5F.9090503@chip-web.com> <443BCE38E921434394B45FF0813FA57850310515@MSGEXSIL1119.ent.wfb.bank.corp> <54D2407B.2060201@devoteam.com> <443BCE38E921434394B45FF0813FA57850310AF2@MSGEXSIL1119.ent.wfb.bank.corp> <443BCE38E921434394B45FF0813FA57850333322@MSGEXSIL1119.ent.wfb.bank.corp> To: user@zookeeper.apache.org X-Mailer: Apple Mail (2.2070.6) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_C167C6B8-0357-4121-A2AB-58F548A2724D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 It's hard to say what the problem is without more information. For = leader election, the servers will try to create a TCP connection with = each other, so if you have tested separately and your sample code worked = for the same ports, the only thing I can think of is that there is a = problem with the server name resolution. Is it possible that the ZK = servers aren't resolving the names of your servers appropriately? Perhaps you could also use tcpdump or some similar tool to inspect the = network traffic. In fact, I think someone here has written a tool to = inspect the leader election traffic and display it. Was it Raul? -Flavio > On 24 Feb 2015, at 16:31, penguin = wrote: >=20 > Ok, I have tried various setups (5 and 3 ZK Server instances), and = neither work, both receiving the =E2=80=98connection refused=E2=80=99 = error. >=20 > I=E2=80=99m now at the 3 ZK ensemble, 1 ZK server instance per = physical machine. >=20 > Myid file is set to 1,2 and 3 respectively on the machines. > Zoo.cfg has the following, and uses 2181 as its client port: >=20 > server.1=3Dservername:2888:3888 > server.2=3Dservername:2888:3888 > server.3=3Dservername:2888:3888 >=20 > Interestingly enough, when I start up the server1 instance, the = zookeeper.out file receives ongoing information on the connection = refused to each server instance. When I start up the servers on = machines 2 and 3, those zookeeper.out files do not receive any = information. Is that normal? >=20 > In any case, I=E2=80=99ve checked the ports by using nc =E2=80=93v = hostname port, and connections are successful. So this is not a = firewall issue or port block issue. I=E2=80=99ve had my OSE=E2=80=99s = check the servers as well to ensure no communications issue. >=20 > *Note, port 2181 is an eForward port. Is this supposed to be used by = Zookeeper? >=20 > Same errors over and over: > 2015-02-24 09:38:00,601 [myid:1] - INFO = [QuorumPeer[myid=3D1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING > 2015-02-24 09:38:00,603 [myid:1] - INFO = [QuorumPeer[myid=3D1]/0.0.0.0:2181:FastLeaderElection@815] - New e > lection. My id =3D 1, proposed zxid=3D0x0 > 2015-02-24 09:38:00,605 [myid:1] - INFO = [WorkerReceiver[myid=3D1]:FastLeaderElection@597] - Notification: > 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), = LOOKING (n.state), 1 (n.sid), 0x0 > (n.peerEpoch) LOOKING (my state) > 2015-02-24 09:38:00,608 [myid:1] - WARN = [WorkerSender[myid=3D1]:QuorumCnxManager@382] - Cannot open channe > l to 2 at election address servername:3888 > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at = java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:33= 9) > at = java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.= java:200) > at = java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)= >=20 > Regards, >=20 > Barry >=20 > From: Flavio Junqueira-2 [via zookeeper-user] = [mailto:ml-node+s578899n7580844h12@n2.nabble.com = ] > Sent: Wednesday, February 04, 2015 4:17 PM > To: Barnett, Barry F > Subject: Re: New Zookeeper Installation - connection refused >=20 > That's because some client is trying to connect to the server, but the = server is looking (not following or leading) and the result of the nc = run confirms it. I think you still have a connectivity problem. There is = something preventing the servers from connecting to each other and it = seems to be something related to your environment. >=20 > -Flavio >=20 >> On 04 Feb 2015, at 17:51, penguin <[hidden = email]> wrote: >>=20 >> I now made this a 3 server ensemble as opposed to a 5 server = ensemble. Now getting different messages in my out logs:P >>=20 >> 2015-02-04 11:48:29,851 [myid:1] - WARN = [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - = Exception causing close of session 0x0 due to java.io.IOException: = ZooKeeperServer not running >> 2015-02-04 11:48:29,851 [myid:1] - INFO = [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed = socket connection for client /xxxxxxxx:64089 (no session established for = client) >>=20 >> $ ./zkServer.sh status >> JMX enabled by default >> Using config: /opt/apps/current/bin/../conf/zoo.cfg >> Error contacting service. It is probably not running. >>=20 >>=20 >> $ echo mntr | nc localhost 2181 >> This ZooKeeper instance is not currently serving requests >>=20 >>=20 >>=20 >> Regards, >>=20 >> Barry >>=20 >>=20 >> From: "J=C3=BCrgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden = email]] >> Sent: Wednesday, February 04, 2015 10:56 AM >> To: Barnett, Barry F >> Subject: Re: New Zookeeper Installation - connection refused >>=20 >> Barry, >> looks like you want to go to the machine where Zk is running and = check >> with "netstat -an" or some other tool on which port Zk is really >> running... unless this is a NAT issue ;-) >>=20 >> Cheers, >> --J=C3=BCrgen >>=20 >> On 04.02.2015 16:51, penguin wrote: >>> $ nc -v remotehostname 2181 >>> nc: connect to remotehostname port 2181 (tcp) failed: Connection = refused >>>=20 >>> Regards, >>>=20 >>> Barry >>>=20 >>=20 >>=20 >> ________________________________ >> If you reply to this email, your message will be added to the = discussion below: >> = http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-conn= ection-refused-tp7580825p7580838.html = >> To unsubscribe from New Zookeeper Installation - connection refused, = click here< >> = NAML >>=20 >>=20 >>=20 >>=20 >> -- >> View this message in context: = http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-conn= ection-refused-tp7580825p7580840.html >> Sent from the zookeeper-user mailing list archive at Nabble.com. >=20 >=20 > ________________________________ > If you reply to this email, your message will be added to the = discussion below: > = http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-conn= ection-refused-tp7580825p7580844.html = > To unsubscribe from New Zookeeper Installation - connection refused, = click = here>. > = NAML> >=20 >=20 >=20 >=20 > -- > View this message in context: = http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-conn= ection-refused-tp7580825p7580875.html = > Sent from the zookeeper-user mailing list archive at Nabble.com = . --Apple-Mail=_C167C6B8-0357-4121-A2AB-58F548A2724D--