Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 303FC175F7 for ; Wed, 11 Mar 2015 15:02:55 +0000 (UTC) Received: (qmail 49521 invoked by uid 500); 11 Mar 2015 15:02:53 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 49485 invoked by uid 500); 11 Mar 2015 15:02:53 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 49455 invoked by uid 500); 11 Mar 2015 15:02:53 -0000 Delivered-To: apmail-hadoop-zookeeper-user@hadoop.apache.org Received: (qmail 49443 invoked by uid 99); 11 Mar 2015 15:02:53 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Mar 2015 15:02:53 +0000 X-ASF-Spam-Status: No, hits=4.5 required=5.0 tests=HTML_MESSAGE,SPF_SOFTFAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of barry.barnett@wellsfargo.com does not designate 162.253.133.43 as permitted sender) Received: from [162.253.133.43] (HELO mwork.nabble.com) (162.253.133.43) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Mar 2015 15:02:26 +0000 Received: from mjim.nabble.com (unknown [162.253.133.84]) by mwork.nabble.com (Postfix) with ESMTP id 9235416EBE7C for ; Wed, 11 Mar 2015 08:02:30 -0700 (PDT) Date: Wed, 11 Mar 2015 08:02:23 -0700 (MST) From: penguin To: zookeeper-user@hadoop.apache.org Message-ID: <443BCE38E921434394B45FF0813FA57850340C61@MSGEXSIL1119.ent.wfb.bank.corp> In-Reply-To: References: <443BCE38E921434394B45FF0813FA5785031021F@MSGEXSIL1119.ent.wfb.bank.corp> <1883587906.2086325.1423056777334.JavaMail.yahoo@mail.yahoo.com> <443BCE38E921434394B45FF0813FA57850310392@MSGEXSIL1119.ent.wfb.bank.corp> <54D23E5F.9090503@chip-web.com> <443BCE38E921434394B45FF0813FA57850310515@MSGEXSIL1119.ent.wfb.bank.corp> <54D2407B.2060201@devoteam.com> <443BCE38E921434394B45FF0813FA57850310AF2@MSGEXSIL1119.ent.wfb.bank.corp> <443BCE38E921434394B45FF0813FA57850333322@MSGEXSIL1119.ent.wfb.bank.corp> Subject: RE: New Zookeeper Installation - connection refused MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_5746_1435579398.1426086143744" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_5746_1435579398.1426086143744 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK ins= tances setup on the same physical machine, so no need to worry about whethe= r multiple physical machines can communicate from one to the other, right? My setup on the Linux sever is as follows: Directory for instance 1: /home/zk1 Directory for instance 2: /home/zk2 Directory for instance 3: /home/zk3 In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2= , zk3 for the other instances): # The number of milliseconds of each tick tickTime=3D2000 # The number of ticks that the initial # synchronization phase can take initLimit=3D10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=3D5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=3D/home/zk1/zookeeper-3.4.6/data dataLogDir=3D/home/zk1/zookeeper-3.4.6/data # the port at which the clients will connect clientPort=3D2181 # the maximum number of client connections. maxClientCnxns=3D60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenanc= e # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3D3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=3D1 server.6=3Dlocalhost:2888:3888 server.7=3Dlocalhost:2889:3889 server.8=3Dlocalhost:2890:3890 the myid files are 6, 7 and 8 respectively for each instance. Is there som= ething I=E2=80=99m missing because I=E2=80=99m still getting the connection= refused warning, and the status of the zk servers doesn=E2=80=99t show any= thing but an error. Zookeeper.out shows: 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=3D6]:QuorumCnxM= anager@38 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketIm= pl.java :339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainS= ocketIm pl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl= .java:1 82) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) zkServer.sh status shows: $ ./zkServer.sh status JMX enabled by default Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg Error contacting service. It is probably not running. $ ./zkServer.sh start JMX enabled by default Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... already running as process 27102. Regards, Barry From: Flavio Junqueira-2 [via zookeeper-user] [mailto:ml-node+s578899n75808= 78h66@n2.nabble.com] Sent: Tuesday, February 24, 2015 6:53 PM To: Barnett, Barry F Subject: Re: New Zookeeper Installation - connection refused It's hard to say what the problem is without more information. For leader e= lection, the servers will try to create a TCP connection with each other, s= o if you have tested separately and your sample code worked for the same po= rts, the only thing I can think of is that there is a problem with the serv= er name resolution. Is it possible that the ZK servers aren't resolving the= names of your servers appropriately? Perhaps you could also use tcpdump or some similar tool to inspect the netw= ork traffic. In fact, I think someone here has written a tool to inspect th= e leader election traffic and display it. Was it Raul? -Flavio > On 24 Feb 2015, at 16:31, penguin <[hidden email]> wrote: > > Ok, I have tried various setups (5 and 3 ZK Server instances), and neithe= r work, both receiving the =E2=80=98connection refused=E2=80=99 error. > > I=E2=80=99m now at the 3 ZK ensemble, 1 ZK server instance per physical m= achine. > > Myid file is set to 1,2 and 3 respectively on the machines. > Zoo.cfg has the following, and uses 2181 as its client port: > > server.1=3Dservername:2888:3888 > server.2=3Dservername:2888:3888 > server.3=3Dservername:2888:3888 > > Interestingly enough, when I start up the server1 instance, the zookeeper= .out file receives ongoing information on the connection refused to each se= rver instance. When I start up the servers on machines 2 and 3, those zook= eeper.out files do not receive any information. Is that normal? > > In any case, I=E2=80=99ve checked the ports by using nc =E2=80=93v hostna= me port, and connections are successful. So this is not a firewall issue o= r port block issue. I=E2=80=99ve had my OSE=E2=80=99s check the servers as= well to ensure no communications issue. > > *Note, port 2181 is an eForward port. Is this supposed to be used by Zoo= keeper? > > Same errors over and over: > 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=3D1]/0.0.0.0:21= 81:QuorumPeer@714] - LOOKING > 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=3D1]/0.0.0.0:21= 81:FastLeaderElection@815] - New e > lection. My id =3D 1, proposed zxid=3D0x0 > 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLe= aderElection@597] - Notification: > 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LO= OKING (n.state), 1 (n.sid), 0x0 > (n.peerEpoch) LOOKING (my state) > 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=3D1]:QuorumCn= xManager@382] - Cannot open channe > l to 2 at election address servername:3888 > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketI= mpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlain= SocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImp= l.java:182) > > Regards, > > Barry > > From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email] >] > Sent: Wednesday, February 04, 2015 4:17 PM > To: Barnett, Barry F > Subject: Re: New Zookeeper Installation - connection refused > > That's because some client is trying to connect to the server, but the se= rver is looking (not following or leading) and the result of the nc run con= firms it. I think you still have a connectivity problem. There is something= preventing the servers from connecting to each other and it seems to be so= mething related to your environment. > > -Flavio > >> On 04 Feb 2015, at 17:51, penguin <[hidden email]> wrote: >> >> I now made this a 3 server ensemble as opposed to a 5 server ensemble. = Now getting different messages in my out logs:P >> >> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0= .0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due= to java.io.IOException: ZooKeeperServer not running >> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0= .0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxx= xxxx:64089 (no session established for client) >> >> $ ./zkServer.sh status >> JMX enabled by default >> Using config: /opt/apps/current/bin/../conf/zoo.cfg >> Error contacting service. It is probably not running. >> >> >> $ echo mntr | nc localhost 2181 >> This ZooKeeper instance is not currently serving requests >> >> >> >> Regards, >> >> Barry >> >> >> From: "J=C3=BCrgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden em= ail]] >> Sent: Wednesday, February 04, 2015 10:56 AM >> To: Barnett, Barry F >> Subject: Re: New Zookeeper Installation - connection refused >> >> Barry, >> looks like you want to go to the machine where Zk is running and check >> with "netstat -an" or some other tool on which port Zk is really >> running... unless this is a NAT issue ;-) >> >> Cheers, >> --J=C3=BCrgen >> >> On 04.02.2015 16:51, penguin wrote: >>> $ nc -v remotehostname 2181 >>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refuse= d >>> >>> Regards, >>> >>> Barry >>> >> >> >> ________________________________ >> If you reply to this email, your message will be added to the discussion= below: >> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-co= nnection-refused-tp7580825p7580838.html >> To unsubscribe from New Zookeeper Installation - connection refused, cli= ck here< >> NAML >> >> >> >> >> -- >> View this message in context: http://zookeeper-user.578899.n2.nabble.com= /New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html >> Sent from the zookeeper-user mailing list archive at Nabble.com. > > > ________________________________ > If you reply to this email, your message will be added to the discussion = below: > http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-con= nection-refused-tp7580825p7580844.html > To unsubscribe from New Zookeeper Installation - connection refused, clic= k here< href=3D"" target=3D"_top" rel=3D"nofollow" link=3D"external"> > NAML> > > > > > -- > View this message in context: http://zookeeper-user.578899.n2.nabble.com/= New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html > Sent from the zookeeper-user mailing list archive at Nabble.com . ________________________________ If you reply to this email, your message will be added to the discussion be= low: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-conne= ction-refused-tp7580825p7580878.html To unsubscribe from New Zookeeper Installation - connection refused, click = here. NAML -- View this message in context: http://zookeeper-user.578899.n2.nabble.com/Ne= w-Zookeeper-Installation-connection-refused-tp7580825p7580904.html Sent from the zookeeper-user mailing list archive at Nabble.com. ------=_Part_5746_1435579398.1426086143744--