Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F1E618851 for ; Wed, 3 Feb 2016 23:40:31 +0000 (UTC) Received: (qmail 14902 invoked by uid 500); 3 Feb 2016 23:40:08 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 14849 invoked by uid 500); 3 Feb 2016 23:40:08 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 14838 invoked by uid 500); 3 Feb 2016 23:40:08 -0000 Delivered-To: apmail-hadoop-zookeeper-user@hadoop.apache.org Received: (qmail 14835 invoked by uid 99); 3 Feb 2016 23:40:08 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2016 23:40:08 +0000 Received: from mail-ig0-f175.google.com (mail-ig0-f175.google.com [209.85.213.175]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 32F8B1A00EA for ; Wed, 3 Feb 2016 23:40:08 +0000 (UTC) Received: by mail-ig0-f175.google.com with SMTP id ik10so1965084igb.1 for ; Wed, 03 Feb 2016 15:40:08 -0800 (PST) X-Gm-Message-State: AG10YOSOtYCygeZ1nW5k65JKQwp4eQGCU/u8jAqTtYFNRI62gPTenTvt06XoUfHWldYyyUQXsd6AjYKlbKDC0A== MIME-Version: 1.0 X-Received: by 10.50.129.67 with SMTP id nu3mr914953igb.48.1454542807523; Wed, 03 Feb 2016 15:40:07 -0800 (PST) Reply-To: cammckenzie@apache.org Received: by 10.107.50.76 with HTTP; Wed, 3 Feb 2016 15:40:07 -0800 (PST) Date: Thu, 4 Feb 2016 10:40:07 +1100 X-Gmail-Original-Message-ID: Message-ID: Subject: ZOOKEEPER-706 From: Cameron McKenzie To: zookeeper-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b41432e5a7408052ae62344 --047d7b41432e5a7408052ae62344 Content-Type: text/plain; charset=UTF-8 Guys, We have an application that potentially has over 100k watches at a given time, and we're running into ZOOKEEPER-706 ( https://issues.apache.org/jira/browse/ZOOKEEPER-706) when reconnecting to the cluster. We tried upgrading the client to 3.4.7 (and upgraded the server also even though looking at the patch I believe it should be a client only fix), but we're still getting the same problem. Logs during reconnection below: 2016-02-04 09:15:36,035 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181 :ZooKeeperServer@617] - Established session 0x152a92db7e80000 with negotiated timeout 10000 for client /10.70.20.57:61918 2016-02-04 09:15:36,193 [myid:1] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x152a92db7e80000 due to java.io.IOException: Len error 19780352 2016-02-04 09:15:36,199 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.70.20.57:61918 which had sessionid 0x152a92db7e80000 2016-02-04 09:15:40,983 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.70.20.57:61922 2016-02-04 09:15:40,984 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x152a92db7e80000 at /10.70.20.57:61922 2016-02-04 09:15:40,984 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x152a92db7e80000 2016-02-04 09:15:40,986 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181 :ZooKeeperServer@617] - Established session 0x152a92db7e80000 with negotiated timeout 10000 for client /10.70.20.57:61922 2016-02-04 09:15:50,713 [myid:1] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x152a92db7e80000 due to java.io.IOException: Len error 959591480 2016-02-04 09:15:50,737 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.70.20.57:61922 which had sessionid 0x152a92db7e80000 2016-02-04 09:16:37,401 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.70.20.57:61934 2016-02-04 09:16:37,411 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x152a92db7e80000 at /10.70.20.57:61934 2016-02-04 09:16:37,412 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x152a92db7e80000 2016-02-04 09:16:37,413 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181 :ZooKeeperServer@617] - Established session 0x152a92db7e80000 with negotiated timeout 10000 for client /10.70.20.57:61934 2016-02-04 09:16:44,187 [myid:1] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@675] - Received packet at server of unknown type 1024 2016-02-04 09:16:44,187 [myid:1] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x152a92db7e80000 due to java.io.IOException: Len error 1293960047 2016-02-04 09:16:44,199 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.70.20.57:61934 which had sessionid 0x152a92db7e80000 2016-02-04 09:16:48,862 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.70.20.57:61938 2016-02-04 09:16:48,870 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x152a92db7e80000 at /10.70.20.57:61938 2016-02-04 09:16:48,870 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x152a92db7e80000 2016-02-04 09:16:48,872 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181 :ZooKeeperServer@617] - Established session 0x152a92db7e80000 with negotiated timeout 10000 for client /10.70.20.57:61938 Any ideas? cheers Cam --047d7b41432e5a7408052ae62344--