Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAL85UBafJt1Y_D5bMm48OneAQWwfPHoSrnj2zfc+V9syeD0dHg@mail.gmail.com>
References: 
 <CAL85UBYqa7NoV_r-Ni4ViDUt1zn9L=X0Hf8KwuhW1x7ff=BYUw@mail.gmail.com>
	<CAL85UBa6cBiNPmk6PkoUZDfXYuznG_WVeqtdqW4VSrNALfvgOg@mail.gmail.com>
	<AD345FD6-545A-4E3B-B223-9E19C7B8A0EB@apache.org>
	<CAL85UBYogrWw_oUaEj8mutEvekJWQ1106_b5cakTV+-UxeX+yQ@mail.gmail.com>
	<CAL85UBa-EKBu+kLzg5K0WPO=8+vU8kGPr7wv+=1iiuyWZ539Dw@mail.gmail.com>
	<CAB5oV28ua69iaZZbz1qNp96=sS5r8PbKM9-ngcSrTTZH_VT-SQ@mail.gmail.com>
	<CAL85UBavr-==ncfwP8ewDNUaStbbJoqZhRsCW6ug69vTqA9swQ@mail.gmail.com>
	<CAL85UBYOMCveSDhqLhCacNXZngsScvbtpADVYt2mBb7pXAZR+g@mail.gmail.com>
	<CAL85UBafJt1Y_D5bMm48OneAQWwfPHoSrnj2zfc+V9syeD0dHg@mail.gmail.com>
Date: Mon, 11 Apr 2016 19:02:10 -0700
Message-ID: 
 <CAL85UBYKDdQrM6zMvZyH7xYTPH2_AU7e+AURr8220Ce7ukHjkw@mail.gmail.com>
Subject: Re: node 2 not rejoining cluster
From: s influxdb <elastic.l.k@gmail.com>
To: user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=001a1148e5e09422380530400cdf

--001a1148e5e09422380530400cdf
Content-Type: text/plain; charset=UTF-8

created a parallel independant zookeeper cluster on the same set of
machines with different ports and that worked. This indicates the port was
the issue.

On Mon, Apr 11, 2016 at 1:35 PM, s influxdb <elastic.l.k@gmail.com> wrote:

> reboot of the server didn't help
>
> On Thu, Apr 7, 2016 at 6:50 PM, s influxdb <elastic.l.k@gmail.com> wrote:
>
>> I ran tcpdump on all the three nodes.
>> It looks like that for every  [PSH, ACK] there is a missing [ACK] from
>> the other nodes to this 2nd node on port 3888.
>>
>>
>> On Thu, Apr 7, 2016 at 1:29 PM, s influxdb <elastic.l.k@gmail.com> wrote:
>>
>>> Thanks Flavio for your quick replies.
>>> The zookeeper version is 3.4.6
>>>
>>>
>>>
>>> On Thu, Apr 7, 2016 at 1:23 PM, Flavio P JUNQUEIRA <fpj@apache.org>
>>> wrote:
>>>
>>>> You need to determine why it is not receiving notification messages.
>>>> From
>>>> the information you've given, it doesn't look like a zookeeper code
>>>> issue.
>>>>
>>>> BTW, which version are you using?
>>>>
>>>> -Flavio
>>>> On 7 Apr 2016 21:20, "s influxdb" <elastic.l.k@gmail.com> wrote:
>>>>
>>>> > nothin on the iptables firewall .
>>>> >
>>>> > What options do i have to reconnect this node to the cluster ?
>>>> >
>>>> >
>>>> > On Thu, Apr 7, 2016 at 10:14 AM, s influxdb <elastic.l.k@gmail.com>
>>>> wrote:
>>>> >
>>>> > > telnet works on 2888 and 3888 to the other nodes. Now i see
>>>> > > java.net.SocketTimeoutException: connect timed out messages in the
>>>> logs
>>>> > for
>>>> > > node 2
>>>> > >
>>>> > > On Thu, Apr 7, 2016 at 3:05 AM, Flavio Junqueira <fpj@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I only see notifications from the node to itself. It says that it
>>>> is
>>>> > >> connected to 1, but it doesn't seem to be receiving the
>>>> notification
>>>> > from
>>>> > >> 1. It also doesn't seem to be receiving the connection request
>>>> from 3.
>>>> > >>
>>>> > >> Last time I've seen something like this was due to iptables rules,
>>>> but
>>>> > if
>>>> > >> it was working before and no configuration has changed, then I
>>>> don't
>>>> > know
>>>> > >> what it could be.
>>>> > >>
>>>> > >> -Flavio
>>>> > >>
>>>> > >> > On 07 Apr 2016, at 05:43, s influxdb <elastic.l.k@gmail.com>
>>>> wrote:
>>>> > >> >
>>>> > >> > this is the pastie
>>>> > >> > http://pastie.org/10788301
>>>> > >> >
>>>> > >> > On Wed, Apr 6, 2016 at 9:41 PM, s influxdb <
>>>> elastic.l.k@gmail.com>
>>>> > >> wrote:
>>>> > >> >
>>>> > >> >> We had one of the node giving OOM java.lang.OutOfMemoryError:
>>>> unable
>>>> > to
>>>> > >> >> create new native thread and then being unresponsive.
>>>> > >> >>
>>>> > >> >> We tried to add the node back to the cluster but with no luck.
>>>> > >> >>
>>>> > >> >> It doesn't seem to "Receive any notification "  messages from
>>>> the
>>>> > other
>>>> > >> >> nodes.
>>>> > >> >> Keeps "Sending notifications " in loop
>>>> > >> >>
>>>> > >> >> Please see attached the logs of the node that is out of
>>>> rotation.
>>>> > >> >>
>>>> > >> >> Any inputs appreciated.
>>>> > >> >>
>>>> > >> >> Thanks
>>>> > >> >>
>>>> > >>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

--001a1148e5e09422380530400cdf--