zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michi Mutsuzaki <mi...@cs.stanford.edu>
Subject Re: New Zookeeper Installation - connection refused
Date Fri, 13 Mar 2015 06:55:02 GMT
Hi Barry,

You don't need to use different client ports if you are running
ZooKeeper servers on separate machines.

On Thu, Mar 12, 2015 at 4:45 AM, penguin <barry.barnett@wellsfargo.com> wrote:
> Ok, the client port change worked for my connection refused issue with instances on the
same server.  I now have 2 followers and 1 leader.  So I've determined that zk can run 3 instances
on the same physical machine with no connection issues.  I was getting the same connection
refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate
an ensemble.  Should I still have different client ports in that scenario?  I had the same
client ports and it was failing miserably.  I am not familiar with zkconf, but I will look
at it.
>
> Regards,
>
> Barry
>
> From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580909h27@n2.nabble.com]
> Sent: Thursday, March 12, 2015 1:26 AM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> Yes that's even better :)
>
> On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>>
wrote:
>
>> Consider zkconf? https://github.com/phunt/zkconf
>>
>> Patrick
>>
>> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>>
wrote:
>>> The clientPort parameter needs to be different for each instance if
>>> you are running them on a same box. Something like this should work:
>>>
>>> https://paste.apache.org/n59c?action=download
>>>
>>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>>
wrote:
>>>> Ok, I have setup my ZooKeeper ensemble on the same server.  I have 3 ZK instances
setup on the same physical machine, so no need to worry about whether multiple physical machines
can communicate from one to the other, right?
>>>> My setup on the Linux sever is as follows:
>>>>
>>>> Directory for instance 1: /home/zk1
>>>> Directory for instance 2: /home/zk2
>>>> Directory for instance 3: /home/zk3
>>>>
>>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2,
zk3 for the other instances):
>>>> # The number of milliseconds of each tick
>>>> tickTime=2000
>>>> # The number of ticks that the initial
>>>> # synchronization phase can take
>>>> initLimit=10
>>>> # The number of ticks that can pass between
>>>> # sending a request and getting an acknowledgement
>>>> syncLimit=5
>>>> # the directory where the snapshot is stored.
>>>> # do not use /tmp for storage, /tmp here is just
>>>> # example sakes.
>>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>>> # the port at which the clients will connect
>>>> clientPort=2181
>>>> # the maximum number of client connections.
>>>> maxClientCnxns=60
>>>> #
>>>> # Be sure to read the maintenance section of the
>>>> # administrator guide before turning on autopurge.
>>>> #
>>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>> #
>>>> # The number of snapshots to retain in dataDir
>>>> #autopurge.snapRetainCount=3
>>>> # Purge task interval in hours
>>>> # Set to "0" to disable auto purge feature
>>>> #autopurge.purgeInterval=1
>>>> server.6=localhost:2888:3888
>>>> server.7=localhost:2889:3889
>>>> server.8=localhost:2890:3890
>>>>
>>>> the myid files are 6, 7 and 8 respectively for each instance.  Is there something
I'm missing because I'm still getting the connection refused warning, and the status of the
zk servers doesn't show anything but an error.
>>>>
>>>> Zookeeper.out shows:
>>>> 2015-03-11 08:59:22,238 [myid:6] - WARN  [WorkerSender[myid=6]:QuorumCnxManager@38
>>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>>> java.net.ConnectException: Connection refused
>>>>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>>> :339)
>>>>         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>>> pl.java:200)
>>>>         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>>> 82)
>>>>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>         at java.net.Socket.connect(Socket.java:579)
>>>>
>>>> zkServer.sh status shows:
>>>>
>>>> $ ./zkServer.sh status
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Error contacting service. It is probably not running.
>>>> $ ./zkServer.sh start
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Starting zookeeper ... already running as process 27102.
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> It's hard to say what the problem is without more information. For leader
election, the servers will try to create a TCP connection with each other, so if you have
tested separately and your sample code worked for the same ports, the only thing I can think
of is that there is a problem with the server name resolution. Is it possible that the ZK
servers aren't resolving the names of your servers appropriately?
>>>>
>>>> Perhaps you could also use tcpdump or some similar tool to inspect the network
traffic. In fact, I think someone here has written a tool to inspect the leader election traffic
and display it. Was it Raul?
>>>>
>>>> -Flavio
>>>>
>>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>>
wrote:
>>>>>
>>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither
work, both receiving the 'connection refused' error.
>>>>>
>>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>>
>>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>>
>>>>> server.1=servername:2888:3888
>>>>> server.2=servername:2888:3888
>>>>> server.3=servername:2888:3888
>>>>>
>>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out
file receives ongoing information on the connection refused to each server instance.  When
I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information.
 Is that normal?
>>>>>
>>>>> In any case, I've checked the ports by using nc -v hostname port, and
connections are successful.  So this is not a firewall issue or port block issue.  I've had
my OSE's check the servers as well to ensure no communications issue.
>>>>>
>>>>> *Note, port 2181 is an eForward port.  Is this supposed to be used by
Zookeeper?
>>>>>
>>>>> Same errors over and over:
>>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714]
- LOOKING
>>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815]
- New e
>>>>> lection. My id =  1, proposed zxid=0x0
>>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@597]
- Notification:
>>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 1 (n.sid), 0x0
>>>>> (n.peerEpoch) LOOKING (my state)
>>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382]
- Cannot open channe
>>>>> l to 2 at election address servername:3888
>>>>> java.net.ConnectException: Connection refused
>>>>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>>        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>>        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1>
<mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> That's because some client is trying to connect to the server, but the
server is looking (not following or leading) and the result of the nc run confirms it. I think
you still have a connectivity problem. There is something preventing the servers from connecting
to each other and it seems to be something related to your environment.
>>>>>
>>>>> -Flavio
>>>>>
>>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>>
wrote:
>>>>>>
>>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble.
 Now getting different messages in my out logs:P
>>>>>>
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362]
- Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007]
- Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>>
>>>>>> $ ./zkServer.sh status
>>>>>> JMX enabled by default
>>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>>> Error contacting service. It is probably not running.
>>>>>>
>>>>>>
>>>>>> $ echo mntr | nc localhost 2181
>>>>>> This ZooKeeper instance is not currently serving requests
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden
email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>>> To: Barnett, Barry F
>>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>>
>>>>>> Barry,
>>>>>> looks like you want to go to the machine where Zk is running and
check
>>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>>> running... unless this is a NAT issue ;-)
>>>>>>
>>>>>> Cheers,
>>>>>> --Jürgen
>>>>>>
>>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>>> $ nc -v remotehostname 2181
>>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection
refused
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> If you reply to this email, your message will be added to the discussion
below:
>>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>>> To unsubscribe from New Zookeeper Installation - connection refused,
click here<
>>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion
below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused,
click here< href="" target="_top" rel="nofollow" link="external">
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion
below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click
here<
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
> To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.

Mime
View raw message