zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From penguin <barry.barn...@wellsfargo.com>
Subject RE: New Zookeeper Installation - connection refused
Date Mon, 16 Mar 2015 13:28:28 GMT
If I use the same zoo.cfg file that I use for the instances running on the same physical machine,
for the remote physical machines zk servers, they fail with connection refused.  All these
machines are in the same network, no firewalls between, no ports blocked.  Very confusing.

Regards,

Barry

From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580914h82@n2.nabble.com]
Sent: Friday, March 13, 2015 2:56 AM
To: Barnett, Barry F
Subject: Re: New Zookeeper Installation - connection refused

Hi Barry,

You don't need to use different client ports if you are running
ZooKeeper servers on separate machines.

On Thu, Mar 12, 2015 at 4:45 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580914&i=0>>
wrote:

> Ok, the client port change worked for my connection refused issue with instances on the
same server.  I now have 2 followers and 1 leader.  So I've determined that zk can run 3 instances
on the same physical machine with no connection issues.  I was getting the same connection
refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate
an ensemble.  Should I still have different client ports in that scenario?  I had the same
client ports and it was failing miserably.  I am not familiar with zkconf, but I will look
at it.
>
> Regards,
>
> Barry
>
> From: Michi Mutsuzaki [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580914&i=1>]
> Sent: Thursday, March 12, 2015 1:26 AM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> Yes that's even better :)
>
> On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>>
wrote:
>
>> Consider zkconf? https://github.com/phunt/zkconf
>>
>> Patrick
>>
>> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>>
wrote:
>>> The clientPort parameter needs to be different for each instance if
>>> you are running them on a same box. Something like this should work:
>>>
>>> https://paste.apache.org/n59c?action=download
>>>
>>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>>
wrote:
>>>> Ok, I have setup my ZooKeeper ensemble on the same server.  I have 3 ZK instances
setup on the same physical machine, so no need to worry about whether multiple physical machines
can communicate from one to the other, right?
>>>> My setup on the Linux sever is as follows:
>>>>
>>>> Directory for instance 1: /home/zk1
>>>> Directory for instance 2: /home/zk2
>>>> Directory for instance 3: /home/zk3
>>>>
>>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2,
zk3 for the other instances):
>>>> # The number of milliseconds of each tick
>>>> tickTime=2000
>>>> # The number of ticks that the initial
>>>> # synchronization phase can take
>>>> initLimit=10
>>>> # The number of ticks that can pass between
>>>> # sending a request and getting an acknowledgement
>>>> syncLimit=5
>>>> # the directory where the snapshot is stored.
>>>> # do not use /tmp for storage, /tmp here is just
>>>> # example sakes.
>>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>>> # the port at which the clients will connect
>>>> clientPort=2181
>>>> # the maximum number of client connections.
>>>> maxClientCnxns=60
>>>> #
>>>> # Be sure to read the maintenance section of the
>>>> # administrator guide before turning on autopurge.
>>>> #
>>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>> #
>>>> # The number of snapshots to retain in dataDir
>>>> #autopurge.snapRetainCount=3
>>>> # Purge task interval in hours
>>>> # Set to "0" to disable auto purge feature
>>>> #autopurge.purgeInterval=1
>>>> server.6=localhost:2888:3888
>>>> server.7=localhost:2889:3889
>>>> server.8=localhost:2890:3890
>>>>
>>>> the myid files are 6, 7 and 8 respectively for each instance.  Is there something
I'm missing because I'm still getting the connection refused warning, and the status of the
zk servers doesn't show anything but an error.
>>>>
>>>> Zookeeper.out shows:
>>>> 2015-03-11 08:59:22,238 [myid:6] - WARN  [WorkerSender[myid=6]:QuorumCnxManager@38
>>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>>> java.net.ConnectException: Connection refused
>>>>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>>> :339)
>>>>         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>>> pl.java:200)
>>>>         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>>> 82)
>>>>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>         at java.net.Socket.connect(Socket.java:579)
>>>>
>>>> zkServer.sh status shows:
>>>>
>>>> $ ./zkServer.sh status
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Error contacting service. It is probably not running.
>>>> $ ./zkServer.sh start
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Starting zookeeper ... already running as process 27102.
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> It's hard to say what the problem is without more information. For leader
election, the servers will try to create a TCP connection with each other, so if you have
tested separately and your sample code worked for the same ports, the only thing I can think
of is that there is a problem with the server name resolution. Is it possible that the ZK
servers aren't resolving the names of your servers appropriately?
>>>>
>>>> Perhaps you could also use tcpdump or some similar tool to inspect the network
traffic. In fact, I think someone here has written a tool to inspect the leader election traffic
and display it. Was it Raul?
>>>>
>>>> -Flavio
>>>>
>>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>>
wrote:
>>>>>
>>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither
work, both receiving the 'connection refused' error.
>>>>>
>>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>>
>>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>>
>>>>> server.1=servername:2888:3888
>>>>> server.2=servername:2888:3888
>>>>> server.3=servername:2888:3888
>>>>>
>>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out
file receives ongoing information on the connection refused to each server instance.  When
I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information.
 Is that normal?
>>>>>
>>>>> In any case, I've checked the ports by using nc -v hostname port, and
connections are successful.  So this is not a firewall issue or port block issue.  I've had
my OSE's check the servers as well to ensure no communications issue.
>>>>>
>>>>> *Note, port 2181 is an eForward port.  Is this supposed to be used by
Zookeeper?
>>>>>
>>>>> Same errors over and over:
>>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714]
- LOOKING
>>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815]
- New e
>>>>> lection. My id =  1, proposed zxid=0x0
>>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@597]
- Notification:
>>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 1 (n.sid), 0x0
>>>>> (n.peerEpoch) LOOKING (my state)
>>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382]
- Cannot open channe
>>>>> l to 2 at election address servername:3888
>>>>> java.net.ConnectException: Connection refused
>>>>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>>        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>>        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1>
<mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2<mailto:[hidden%20email]%3c/user/SendEmail.jtp?type=node&node=7580878&i=2>>>]
>>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> That's because some client is trying to connect to the server, but the
server is looking (not following or leading) and the result of the nc run confirms it. I think
you still have a connectivity problem. There is something preventing the servers from connecting
to each other and it seems to be something related to your environment.
>>>>>
>>>>> -Flavio
>>>>>
>>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>>
wrote:
>>>>>>
>>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble.
 Now getting different messages in my out logs:P
>>>>>>
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362]
- Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007]
- Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>>
>>>>>> $ ./zkServer.sh status
>>>>>> JMX enabled by default
>>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>>> Error contacting service. It is probably not running.
>>>>>>
>>>>>>
>>>>>> $ echo mntr | nc localhost 2181
>>>>>> This ZooKeeper instance is not currently serving requests
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden
email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>>> To: Barnett, Barry F
>>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>>
>>>>>> Barry,
>>>>>> looks like you want to go to the machine where Zk is running and
check
>>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>>> running... unless this is a NAT issue ;-)
>>>>>>
>>>>>> Cheers,
>>>>>> --Jürgen
>>>>>>
>>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>>> $ nc -v remotehostname 2181
>>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection
refused
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> If you reply to this email, your message will be added to the discussion
below:
>>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>>> To unsubscribe from New Zookeeper Installation - connection refused,
click here<
>>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion
below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused,
click here< href="" target="_top" rel="nofollow" link="external">
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion
below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click
here<
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
> To unsubscribe from New Zookeeper Installation - connection refused, click here<
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580914.html
To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580921.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message