Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 40430198E6 for ; Thu, 7 Apr 2016 17:14:58 +0000 (UTC) Received: (qmail 51915 invoked by uid 500); 7 Apr 2016 17:14:57 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 51863 invoked by uid 500); 7 Apr 2016 17:14:57 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 51779 invoked by uid 99); 7 Apr 2016 17:14:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2016 17:14:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CC8EDC1256 for ; Thu, 7 Apr 2016 17:14:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id J8Ey3iEZLb5R for ; Thu, 7 Apr 2016 17:14:54 +0000 (UTC) Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 3B15C5FB0C for ; Thu, 7 Apr 2016 17:14:54 +0000 (UTC) Received: by mail-wm0-f47.google.com with SMTP id l6so34406558wml.1 for ; Thu, 07 Apr 2016 10:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=p6lgcJ3y6Wq/wdMAcyonCtQ0gXYhhRocEE4NAKhnlCg=; b=HAAe2G8+B8mvautfw+x0mscf840snLnPOYicqkO0NlMeixmIZUMGYnegAr7ENAIESz mWC50Ig/+UuHisMdtuOC0dBkNmWMglddFJYJGduVzBp3c6oKJmFBGHVqrd4nH7JzUdKq RygoEE15XuBJWuE7GTCYbaPq9DlsiFPbPQLbMHIu5vK2PqkepZ6cuq3Wlw9GoR9jPjvH FZ3HF10ephb1BQm0+hHLkY4Q4kEyeAsYZT9COZpeWbH1otzmEHLtbAM/02xL7Qj6LoPN b2gVdBBxET60HWMaGC4cqjGD0rXm/5gvE3lWdKyVTsMgxAPTgl0HTyCf8Vv5rRAIbzYg lwJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=p6lgcJ3y6Wq/wdMAcyonCtQ0gXYhhRocEE4NAKhnlCg=; b=fDIBWacLEn6LPVP5R7wJNljDCYd11uGCxus/6YBjd7eqbniMC7pNP12EUhTiPhkGFY NmFf8hctsxQN3OD1gw5b8Q6olEYh/zy4DHRNpwGUu10jljpFr6SWgh77aPRY3yN5YhG6 TUpEf7he+ARR6la1KMvBil1NnVFJ9EBFW4rorc/cJvRo9cJurRYvtxeAvLBKiIRmFWX6 w7bEBlWDDbXshI7mjVYXqJX99ggOIt/x5s8OEMobbM4re+h7ZMVM6QCRxifoJs8YyG2L An1yweCl4m63NfQmB0Afwi1pCelpAl/kjA5ofI1hr28PMLueQ14CPyxz/Je57D2Wlfq2 gcUw== X-Gm-Message-State: AD7BkJLesixdLi2bDNyZVmGS5g9adDYKF0mHwTx5ClNRxBmR0WquY44PDoJCh7tuLHs6FKl3Cdg5k3avw8qlzQ== MIME-Version: 1.0 X-Received: by 10.194.22.68 with SMTP id b4mr4609228wjf.158.1460049293914; Thu, 07 Apr 2016 10:14:53 -0700 (PDT) Received: by 10.28.211.10 with HTTP; Thu, 7 Apr 2016 10:14:53 -0700 (PDT) In-Reply-To: References: Date: Thu, 7 Apr 2016 10:14:53 -0700 Message-ID: Subject: Re: node 2 not rejoining cluster From: s influxdb To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary=047d7b5d392a84b909052fe837b0 --047d7b5d392a84b909052fe837b0 Content-Type: text/plain; charset=UTF-8 telnet works on 2888 and 3888 to the other nodes. Now i see java.net.SocketTimeoutException: connect timed out messages in the logs for node 2 On Thu, Apr 7, 2016 at 3:05 AM, Flavio Junqueira wrote: > I only see notifications from the node to itself. It says that it is > connected to 1, but it doesn't seem to be receiving the notification from > 1. It also doesn't seem to be receiving the connection request from 3. > > Last time I've seen something like this was due to iptables rules, but if > it was working before and no configuration has changed, then I don't know > what it could be. > > -Flavio > > > On 07 Apr 2016, at 05:43, s influxdb wrote: > > > > this is the pastie > > http://pastie.org/10788301 > > > > On Wed, Apr 6, 2016 at 9:41 PM, s influxdb > wrote: > > > >> We had one of the node giving OOM java.lang.OutOfMemoryError: unable to > >> create new native thread and then being unresponsive. > >> > >> We tried to add the node back to the cluster but with no luck. > >> > >> It doesn't seem to "Receive any notification " messages from the other > >> nodes. > >> Keeps "Sending notifications " in loop > >> > >> Please see attached the logs of the node that is out of rotation. > >> > >> Any inputs appreciated. > >> > >> Thanks > >> > > --047d7b5d392a84b909052fe837b0--