Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 885F7200B8B for ; Tue, 4 Oct 2016 09:13:22 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 87025160AC9; Tue, 4 Oct 2016 07:13:22 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 99538160AC5 for ; Tue, 4 Oct 2016 09:13:21 +0200 (CEST) Received: (qmail 81600 invoked by uid 500); 4 Oct 2016 07:13:20 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 81587 invoked by uid 99); 4 Oct 2016 07:13:20 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Oct 2016 07:13:20 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 79E8C2C044E for ; Tue, 4 Oct 2016 07:13:20 +0000 (UTC) Date: Tue, 4 Oct 2016 07:13:20 +0000 (UTC) From: "Alex Ivanov (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-13653) ZKDelegationTokenSecretManager curator client seems to rapidly connect & disconnect from ZK MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 04 Oct 2016 07:13:22 -0000 [ https://issues.apache.org/jira/browse/HADOOP-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15544588#comment-15544588 ] Alex Ivanov commented on HADOOP-13653: -------------------------------------- [~xiaochen], I wonder if the issue we've seen is related to the curator version used by KMS. I see a very similar exception stack to the one here: [CURATOR-209|https://issues.apache.org/jira/browse/CURATOR-209]. Do you see any problem with having KMS use curator 2.10 instead? > ZKDelegationTokenSecretManager curator client seems to rapidly connect & disconnect from ZK > ------------------------------------------------------------------------------------------- > > Key: HADOOP-13653 > URL: https://issues.apache.org/jira/browse/HADOOP-13653 > Project: Hadoop Common > Issue Type: Bug > Components: kms > Reporter: Alex Ivanov > Priority: Critical > > During periods of time, KMS gets in a connect/disconnect loop from Zookeeper. It is not clear what causes the connection to be closed. I didn't see any issues on the ZK server side, so the issue must reside on client side. > *Example errors* > NOTE: I had to filter the logs heavily since they were many GB in size (thanks to curator error logging). What is left is an illustration of the delegation token creations, and the Zookeeper sessions getting lost and re-established over the course of 2 hours. > {code} > 2016-09-25 01:43:04,377 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [75027a21ab399aa7789d6907d70fadc4, 46] > 2016-09-25 01:43:04,557 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [1106d0754d43dcf29324d7be737f51f0, 46] > 2016-09-25 01:43:11,846 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [4426092c861f49c6ba0c60b49b9539e5, 46] > 2016-09-25 01:43:48,974 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [a99efff2705d6489deb059098f18818f, 46] > 2016-09-25 01:43:49,174 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [398b5962fd647880961ba5e86a77b414, 46] > 2016-09-25 01:44:03,359 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [413187e62a21b5459422b5c524315d06, 46] > 2016-09-25 01:44:03,625 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [7cc2c0d82edd40e7e6f6f40af20d04d3, 46] > 2016-09-25 01:44:06,062 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [bd9394fce20607c12bc00104bea49284, 46] > 2016-09-25 01:44:07,134 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [7dad3bd10526517e5e1cfccd2e96074a, 46] > 2016-09-25 01:44:07,230 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [a712ed40687580647d070c9c7f525e15, 46] > 2016-09-25 01:44:48,481 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [44bfefa31192c68e3cc053eec4e57e14, 46] > 2016-09-25 01:44:48,522 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [67efc2aa65eeba701ad7d3d7bab51def, 46] > 2016-09-25 01:44:50,259 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [b43e641f58dfbd2c72550ab6804f37d1, 46] > 2016-09-25 01:44:54,271 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [ac2fbcf404c633759b75e6d6aae00e05, 46] > 2016-09-25 01:44:56,141 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [cdbd224079a4a10400d00d0b8eece008, 46] > 2016-09-25 01:45:01,328 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [e03218f4835524f3d05519d27bb04e35, 46] > 2016-09-25 01:45:02,728 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [569ae6d666d584b6843fffc47a63d147, 46] > 2016-09-25 01:45:02,832 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [c9048271483da234c12f75569b9513c6, 46] > 2016-09-25 01:45:05,536 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [f519d621389e41b63e8d92b4cb15f832, 46] > 2016-09-25 01:45:07,886 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [45cf6ba58b2bb348ac5e88fa18fe9dad, 46] > 2016-09-25 01:47:24,346 WARN ConnectionState - Connection attempt unsuccessful after 66294 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. > 2016-09-25 01:47:25,120 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [f160a865db69ef33548f146c9b3b84c6, 46] > 2016-09-25 01:47:25,276 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [9d60add471464e01ef691c43bd901d96, 46] > 2016-09-25 01:47:28,739 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [659ecabf02ff809736202a8484ff2be8, 46] > 2016-09-25 01:48:33,233 WARN ConnectionState - Connection attempt unsuccessful after 64494 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. > 2016-09-25 01:48:33,306 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [15b87dd6c2251177d8db9dda415d0e06, 46] > 2016-09-25 01:48:33,459 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [306aa796017aab2b559bf503f81175e0, 46] > 2016-09-25 01:48:34,665 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [6c47cc10edcf0931e7e26665f99dadc5, 46] > 2016-09-25 01:49:40,669 WARN ConnectionState - Connection attempt unsuccessful after 66006 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. > 2016-09-25 01:49:40,847 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [049585962d9ac1cdb4df17f826891130, 46] > 2016-09-25 01:49:41,523 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [a0fd8cffe2ba40d63d4cd009aabb77bb, 46] > 2016-09-25 01:49:44,811 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [ea94d6cc4df6621a38be9f79121b5cc6, 46] > 2016-09-25 01:50:48,781 WARN ConnectionState - Connection attempt unsuccessful after 64005 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. > ... > ... > 2016-09-25 03:39:51,312 WARN ConnectionState - Connection attempt unsuccessful after 60001 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. > 2016-09-25 03:39:51,752 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [2b605504535c741de3d18ed30fe90d7c, 46] > 2016-09-25 03:39:55,345 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [9dda97d42df3bf8ae805f1da2857bc33, 46] > 2016-09-25 03:39:55,668 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [f52dcf38a4565a904be93fe3de92825d, 46] > 2016-09-25 03:41:02,660 WARN ConnectionState - Connection attempt unsuccessful after 67005 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. > 2016-09-25 03:41:02,921 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [770057f6b1412f096a92ce6599c6422a, 46] > 2016-09-25 03:41:03,035 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [f2b366612a15e0a450eac8b4cf556515, 46] > 2016-09-25 03:41:05,903 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [a3b2d0684e1373fc2f9e5c8bff5e9939, 46] > 2016-09-25 03:41:05,994 INFO AbstractDelegationTokenSecretManager - Creating password for identifier: [f9e72fd48c8aab0a737d2cc8319aa389, 46] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org