Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3696217F84 for ; Thu, 12 Mar 2015 05:27:17 +0000 (UTC) Received: (qmail 97981 invoked by uid 500); 12 Mar 2015 05:27:16 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 97930 invoked by uid 500); 12 Mar 2015 05:27:16 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 97918 invoked by uid 99); 12 Mar 2015 05:27:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2015 05:27:16 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mutsuzaki@gmail.com designates 209.85.223.180 as permitted sender) Received: from [209.85.223.180] (HELO mail-ie0-f180.google.com) (209.85.223.180) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2015 05:26:51 +0000 Received: by iegc3 with SMTP id c3so19067492ieg.3 for ; Wed, 11 Mar 2015 22:24:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=776kSbqzdCB44QHYzWWdJYj3yKtnzNXlbS+hEeGtc+A=; b=TmxRuEbROfHRFAiR55va9hI/ciqFlBGZqORReHGgw6ftfuTYK/TcK4PW8BRT8NzK0D Jo+0HjPrYg47/EU1rpdpG/I7oZrQunxf9xcXPnXHNL1rtTDXN8eup7WCqQWViYf5uPMc wJsOvQxpnjtrzETq/u7tRXzqfuItLnlIp0vvVb7ns28aWFHdIFAyIM9hnfrFbgzPq293 jkdg4JQXLZ7eSNG5LlZRWICODR2+YjLxP9lxJLuj0Hpu0LsvnbwZ7H9UkTGpqPPtwOhT urejRw80PGc+Tg8ufgOj2gxTAFE4FD75mFNHbh76APipytAhE3tTQXJIq45XaSEuiLqj oqeg== MIME-Version: 1.0 X-Received: by 10.202.175.76 with SMTP id y73mr8448141oie.81.1426137874456; Wed, 11 Mar 2015 22:24:34 -0700 (PDT) Reply-To: michi@cs.stanford.edu Sender: mutsuzaki@gmail.com Received: by 10.202.91.2 with HTTP; Wed, 11 Mar 2015 22:24:34 -0700 (PDT) In-Reply-To: References: <443BCE38E921434394B45FF0813FA5785031021F@MSGEXSIL1119.ent.wfb.bank.corp> <1883587906.2086325.1423056777334.JavaMail.yahoo@mail.yahoo.com> <443BCE38E921434394B45FF0813FA57850310392@MSGEXSIL1119.ent.wfb.bank.corp> <54D23E5F.9090503@chip-web.com> <443BCE38E921434394B45FF0813FA57850310515@MSGEXSIL1119.ent.wfb.bank.corp> <54D2407B.2060201@devoteam.com> <443BCE38E921434394B45FF0813FA57850310AF2@MSGEXSIL1119.ent.wfb.bank.corp> <443BCE38E921434394B45FF0813FA57850333322@MSGEXSIL1119.ent.wfb.bank.corp> <443BCE38E921434394B45FF0813FA57850340C61@MSGEXSIL1119.ent.wfb.bank.corp> Date: Wed, 11 Mar 2015 22:24:34 -0700 X-Google-Sender-Auth: UNH4ho-x-KDCdlnP8lU0Nqq37fk Message-ID: Subject: Re: New Zookeeper Installation - connection refused From: Michi Mutsuzaki To: "user@zookeeper.apache.org" Cc: "zookeeper-user@hadoop.apache.org" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Yes that's even better :) On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt wrote: > Consider zkconf? https://github.com/phunt/zkconf > > Patrick > > On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki = wrote: >> The clientPort parameter needs to be different for each instance if >> you are running them on a same box. Something like this should work: >> >> https://paste.apache.org/n59c?action=3Ddownload >> >> On Wed, Mar 11, 2015 at 8:02 AM, penguin = wrote: >>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK= instances setup on the same physical machine, so no need to worry about wh= ether multiple physical machines can communicate from one to the other, rig= ht? >>> My setup on the Linux sever is as follows: >>> >>> Directory for instance 1: /home/zk1 >>> Directory for instance 2: /home/zk2 >>> Directory for instance 3: /home/zk3 >>> >>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with= zk2, zk3 for the other instances): >>> # The number of milliseconds of each tick >>> tickTime=3D2000 >>> # The number of ticks that the initial >>> # synchronization phase can take >>> initLimit=3D10 >>> # The number of ticks that can pass between >>> # sending a request and getting an acknowledgement >>> syncLimit=3D5 >>> # the directory where the snapshot is stored. >>> # do not use /tmp for storage, /tmp here is just >>> # example sakes. >>> dataDir=3D/home/zk1/zookeeper-3.4.6/data >>> dataLogDir=3D/home/zk1/zookeeper-3.4.6/data >>> # the port at which the clients will connect >>> clientPort=3D2181 >>> # the maximum number of client connections. >>> maxClientCnxns=3D60 >>> # >>> # Be sure to read the maintenance section of the >>> # administrator guide before turning on autopurge. >>> # >>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_mainte= nance >>> # >>> # The number of snapshots to retain in dataDir >>> #autopurge.snapRetainCount=3D3 >>> # Purge task interval in hours >>> # Set to "0" to disable auto purge feature >>> #autopurge.purgeInterval=3D1 >>> server.6=3Dlocalhost:2888:3888 >>> server.7=3Dlocalhost:2889:3889 >>> server.8=3Dlocalhost:2890:3890 >>> >>> the myid files are 6, 7 and 8 respectively for each instance. Is there= something I'm missing because I'm still getting the connection refused war= ning, and the status of the zk servers doesn't show anything but an error. >>> >>> Zookeeper.out shows: >>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=3D6]:Quorum= CnxManager@38 >>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3= 890 >>> java.net.ConnectException: Connection refused >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock= etImpl.java >>> :339) >>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPl= ainSocketIm >>> pl.java:200) >>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket= Impl.java:1 >>> 82) >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>> at java.net.Socket.connect(Socket.java:579) >>> >>> zkServer.sh status shows: >>> >>> $ ./zkServer.sh status >>> JMX enabled by default >>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg >>> Error contacting service. It is probably not running. >>> $ ./zkServer.sh start >>> JMX enabled by default >>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg >>> Starting zookeeper ... already running as process 27102. >>> >>> Regards, >>> >>> Barry >>> >>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:ml-node+s578899n7= 580878h66@n2.nabble.com] >>> Sent: Tuesday, February 24, 2015 6:53 PM >>> To: Barnett, Barry F >>> Subject: Re: New Zookeeper Installation - connection refused >>> >>> It's hard to say what the problem is without more information. For lead= er election, the servers will try to create a TCP connection with each othe= r, so if you have tested separately and your sample code worked for the sam= e ports, the only thing I can think of is that there is a problem with the = server name resolution. Is it possible that the ZK servers aren't resolving= the names of your servers appropriately? >>> >>> Perhaps you could also use tcpdump or some similar tool to inspect the = network traffic. In fact, I think someone here has written a tool to inspec= t the leader election traffic and display it. Was it Raul? >>> >>> -Flavio >>> >>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]> wrote: >>>> >>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and nei= ther work, both receiving the 'connection refused' error. >>>> >>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machin= e. >>>> >>>> Myid file is set to 1,2 and 3 respectively on the machines. >>>> Zoo.cfg has the following, and uses 2181 as its client port: >>>> >>>> server.1=3Dservername:2888:3888 >>>> server.2=3Dservername:2888:3888 >>>> server.3=3Dservername:2888:3888 >>>> >>>> Interestingly enough, when I start up the server1 instance, the zookee= per.out file receives ongoing information on the connection refused to each= server instance. When I start up the servers on machines 2 and 3, those z= ookeeper.out files do not receive any information. Is that normal? >>>> >>>> In any case, I've checked the ports by using nc -v hostname port, and = connections are successful. So this is not a firewall issue or port block = issue. I've had my OSE's check the servers as well to ensure no communicat= ions issue. >>>> >>>> *Note, port 2181 is an eForward port. Is this supposed to be used by = Zookeeper? >>>> >>>> Same errors over and over: >>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=3D1]/0.0.0.0= :2181:QuorumPeer@714] - LOOKING >>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=3D1]/0.0.0.0= :2181:FastLeaderElection@815] - New e >>>> lection. My id =3D 1, proposed zxid=3D0x0 >>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=3D1]:Fas= tLeaderElection@597] - Notification: >>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round),= LOOKING (n.state), 1 (n.sid), 0x0 >>>> (n.peerEpoch) LOOKING (my state) >>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=3D1]:Quoru= mCnxManager@382] - Cannot open channe >>>> l to 2 at election address servername:3888 >>>> java.net.ConnectException: Connection refused >>>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock= etImpl.java:339) >>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPl= ainSocketImpl.java:200) >>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket= Impl.java:182) >>>> >>>> Regards, >>>> >>>> Barry >>>> >>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email] >] >>>> Sent: Wednesday, February 04, 2015 4:17 PM >>>> To: Barnett, Barry F >>>> Subject: Re: New Zookeeper Installation - connection refused >>>> >>>> That's because some client is trying to connect to the server, but the= server is looking (not following or leading) and the result of the nc run = confirms it. I think you still have a connectivity problem. There is someth= ing preventing the servers from connecting to each other and it seems to be= something related to your environment. >>>> >>>> -Flavio >>>> >>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]> wrote: >>>>> >>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble= . Now getting different messages in my out logs:P >>>>> >>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.= 0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 = due to java.io.IOException: ZooKeeperServer not running >>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.= 0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /x= xxxxxxx:64089 (no session established for client) >>>>> >>>>> $ ./zkServer.sh status >>>>> JMX enabled by default >>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg >>>>> Error contacting service. It is probably not running. >>>>> >>>>> >>>>> $ echo mntr | nc localhost 2181 >>>>> This ZooKeeper instance is not currently serving requests >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Barry >>>>> >>>>> >>>>> From: "J=FCrgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden em= ail]] >>>>> Sent: Wednesday, February 04, 2015 10:56 AM >>>>> To: Barnett, Barry F >>>>> Subject: Re: New Zookeeper Installation - connection refused >>>>> >>>>> Barry, >>>>> looks like you want to go to the machine where Zk is running and chec= k >>>>> with "netstat -an" or some other tool on which port Zk is really >>>>> running... unless this is a NAT issue ;-) >>>>> >>>>> Cheers, >>>>> --J=FCrgen >>>>> >>>>> On 04.02.2015 16:51, penguin wrote: >>>>>> $ nc -v remotehostname 2181 >>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection ref= used >>>>>> >>>>>> Regards, >>>>>> >>>>>> Barry >>>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> If you reply to this email, your message will be added to the discuss= ion below: >>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation= -connection-refused-tp7580825p7580838.html >>>>> To unsubscribe from New Zookeeper Installation - connection refused, = click here< >>>>> NAML >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.= com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html >>>>> Sent from the zookeeper-user mailing list archive at Nabble.com. >>>> >>>> >>>> ________________________________ >>>> If you reply to this email, your message will be added to the discussi= on below: >>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-= connection-refused-tp7580825p7580844.html >>>> To unsubscribe from New Zookeeper Installation - connection refused, c= lick here< href=3D"" target=3D"_top" rel=3D"nofollow" link=3D"external"> >>>> NAML> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://zookeeper-user.578899.n2.nabble.c= om/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html >>>> Sent from the zookeeper-user mailing list archive at Nabble.com . >>> >>> >>> ________________________________ >>> If you reply to this email, your message will be added to the discussio= n below: >>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-c= onnection-refused-tp7580825p7580878.html >>> To unsubscribe from New Zookeeper Installation - connection refused, cl= ick here. >>> NAML >>> >>> >>> >>> >>> -- >>> View this message in context: http://zookeeper-user.578899.n2.nabble.co= m/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html >>> Sent from the zookeeper-user mailing list archive at Nabble.com.