zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From penguin <barry.barn...@wellsfargo.com>
Subject RE: New Zookeeper Installation - connection refused
Date Thu, 12 Mar 2015 11:45:03 GMT
Ok, the client port change worked for my connection refused issue with instances on the same
server.  I now have 2 followers and 1 leader.  So I’ve determined that zk can run 3 instances
on the same physical machine with no connection issues.  I was getting the same connection
refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate
an ensemble.  Should I still have different client ports in that scenario?  I had the same
client ports and it was failing miserably.  I am not familiar with zkconf, but I will look
at it.

Regards,

Barry

From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580909h27@n2.nabble.com]
Sent: Thursday, March 12, 2015 1:26 AM
To: Barnett, Barry F
Subject: Re: New Zookeeper Installation - connection refused

Yes that's even better :)

On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>>
wrote:

> Consider zkconf? https://github.com/phunt/zkconf
>
> Patrick
>
> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>>
wrote:
>> The clientPort parameter needs to be different for each instance if
>> you are running them on a same box. Something like this should work:
>>
>> https://paste.apache.org/n59c?action=download
>>
>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>>
wrote:
>>> Ok, I have setup my ZooKeeper ensemble on the same server.  I have 3 ZK instances
setup on the same physical machine, so no need to worry about whether multiple physical machines
can communicate from one to the other, right?
>>> My setup on the Linux sever is as follows:
>>>
>>> Directory for instance 1: /home/zk1
>>> Directory for instance 2: /home/zk2
>>> Directory for instance 3: /home/zk3
>>>
>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2,
zk3 for the other instances):
>>> # The number of milliseconds of each tick
>>> tickTime=2000
>>> # The number of ticks that the initial
>>> # synchronization phase can take
>>> initLimit=10
>>> # The number of ticks that can pass between
>>> # sending a request and getting an acknowledgement
>>> syncLimit=5
>>> # the directory where the snapshot is stored.
>>> # do not use /tmp for storage, /tmp here is just
>>> # example sakes.
>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>> # the port at which the clients will connect
>>> clientPort=2181
>>> # the maximum number of client connections.
>>> maxClientCnxns=60
>>> #
>>> # Be sure to read the maintenance section of the
>>> # administrator guide before turning on autopurge.
>>> #
>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>> #
>>> # The number of snapshots to retain in dataDir
>>> #autopurge.snapRetainCount=3
>>> # Purge task interval in hours
>>> # Set to "0" to disable auto purge feature
>>> #autopurge.purgeInterval=1
>>> server.6=localhost:2888:3888
>>> server.7=localhost:2889:3889
>>> server.8=localhost:2890:3890
>>>
>>> the myid files are 6, 7 and 8 respectively for each instance.  Is there something
I'm missing because I'm still getting the connection refused warning, and the status of the
zk servers doesn't show anything but an error.
>>>
>>> Zookeeper.out shows:
>>> 2015-03-11 08:59:22,238 [myid:6] - WARN  [WorkerSender[myid=6]:QuorumCnxManager@38
>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>> java.net.ConnectException: Connection refused
>>>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>> :339)
>>>         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>> pl.java:200)
>>>         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>> 82)
>>>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>         at java.net.Socket.connect(Socket.java:579)
>>>
>>> zkServer.sh status shows:
>>>
>>> $ ./zkServer.sh status
>>> JMX enabled by default
>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>> Error contacting service. It is probably not running.
>>> $ ./zkServer.sh start
>>> JMX enabled by default
>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>> Starting zookeeper ... already running as process 27102.
>>>
>>> Regards,
>>>
>>> Barry
>>>
>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>> To: Barnett, Barry F
>>> Subject: Re: New Zookeeper Installation - connection refused
>>>
>>> It's hard to say what the problem is without more information. For leader election,
the servers will try to create a TCP connection with each other, so if you have tested separately
and your sample code worked for the same ports, the only thing I can think of is that there
is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving
the names of your servers appropriately?
>>>
>>> Perhaps you could also use tcpdump or some similar tool to inspect the network
traffic. In fact, I think someone here has written a tool to inspect the leader election traffic
and display it. Was it Raul?
>>>
>>> -Flavio
>>>
>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>>
wrote:
>>>>
>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither
work, both receiving the 'connection refused' error.
>>>>
>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>
>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>
>>>> server.1=servername:2888:3888
>>>> server.2=servername:2888:3888
>>>> server.3=servername:2888:3888
>>>>
>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out
file receives ongoing information on the connection refused to each server instance.  When
I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information.
 Is that normal?
>>>>
>>>> In any case, I've checked the ports by using nc -v hostname port, and connections
are successful.  So this is not a firewall issue or port block issue.  I've had my OSE's check
the servers as well to ensure no communications issue.
>>>>
>>>> *Note, port 2181 is an eForward port.  Is this supposed to be used by Zookeeper?
>>>>
>>>> Same errors over and over:
>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714]
- LOOKING
>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815]
- New e
>>>> lection. My id =  1, proposed zxid=0x0
>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@597]
- Notification:
>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING
(n.state), 1 (n.sid), 0x0
>>>> (n.peerEpoch) LOOKING (my state)
>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382]
- Cannot open channe
>>>> l to 2 at election address servername:3888
>>>> java.net.ConnectException: Connection refused
>>>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1>
<mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> That's because some client is trying to connect to the server, but the server
is looking (not following or leading) and the result of the nc run confirms it. I think you
still have a connectivity problem. There is something preventing the servers from connecting
to each other and it seems to be something related to your environment.
>>>>
>>>> -Flavio
>>>>
>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>>
wrote:
>>>>>
>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble.
 Now getting different messages in my out logs:P
>>>>>
>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362]
- Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007]
- Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>
>>>>> $ ./zkServer.sh status
>>>>> JMX enabled by default
>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>> Error contacting service. It is probably not running.
>>>>>
>>>>>
>>>>> $ echo mntr | nc localhost 2181
>>>>> This ZooKeeper instance is not currently serving requests
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> Barry,
>>>>> looks like you want to go to the machine where Zk is running and check
>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>> running... unless this is a NAT issue ;-)
>>>>>
>>>>> Cheers,
>>>>> --Jürgen
>>>>>
>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>> $ nc -v remotehostname 2181
>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection
refused
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion
below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused,
click here<
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion
below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click
here< href="" target="_top" rel="nofollow" link="external">
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html
<http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>> Sent from the zookeeper-user mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message