Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4F91A111EC for ; Tue, 23 Sep 2014 06:28:31 +0000 (UTC) Received: (qmail 91069 invoked by uid 500); 23 Sep 2014 06:28:27 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 91004 invoked by uid 500); 23 Sep 2014 06:28:27 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 90992 invoked by uid 99); 23 Sep 2014 06:28:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Sep 2014 06:28:27 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tobeg3oogle@gmail.com designates 209.85.215.43 as permitted sender) Received: from [209.85.215.43] (HELO mail-la0-f43.google.com) (209.85.215.43) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Sep 2014 06:28:21 +0000 Received: by mail-la0-f43.google.com with SMTP id gi9so8114261lab.30 for ; Mon, 22 Sep 2014 23:28:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ui6rQhwNMAocIij4eKg5X96wpjY/Qu0bI4IiIVClYLo=; b=wBjQBHGrKSWDWUcskVWvVZetM33hVNowM9BqjHJNZT9jClIx9qzKFixSsZY6Y3QFry TZhv9LYuadKx7UBKwpCVJpo50/+T+ZKY4e04LTmK9vqL4JsIkmRPvvGgoehVTSPJcdLV l9Pxc07G95BKrSBWOraV74Xav/LJk//oYjjUmQLg/sGZwGdPE1vRTbcIMPNxQo/ZF53f Y5E4xX34V4E3aGsfaCrPyAJk0B7A0ETKqeVKjQMbwyCSZDAS3ixs2wJ2/sr7JsRdXEjF AzkbXRjMupybdFp0PqUz058DB81V1yoqm1wo8StYf+iBqE5UWGPcwEBdBPNCbWNQHP6k zi5A== MIME-Version: 1.0 X-Received: by 10.153.11.132 with SMTP id ei4mr30545877lad.24.1411453679806; Mon, 22 Sep 2014 23:27:59 -0700 (PDT) Received: by 10.25.25.149 with HTTP; Mon, 22 Sep 2014 23:27:59 -0700 (PDT) In-Reply-To: References: <1411191887.72472.YahooMailNeo@web140602.mail.bf1.yahoo.com> Date: Tue, 23 Sep 2014 14:27:59 +0800 Message-ID: Subject: Re: HBase establishes session with ZooKeeper and close the session immediately From: tobe To: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=001a1134787a33b08e0503b5ab06 X-Virus-Checked: Checked by ClamAV on apache.org --001a1134787a33b08e0503b5ab06 Content-Type: text/plain; charset=UTF-8 I mean I have no idea why the RegionServer has so many log about establishing and closing session. I don't think we have so many clients within every second. This happens when I restart the server but not for all HBase clusters. On Tue, Sep 23, 2014 at 11:57 AM, Qiang Tian wrote: > sorry, what reason do you mean? > I took a quick look at 0.94.21, it looks only client code paths call below > constructor, so this version looks immune to the problem.. > > public CatalogTracker(final Configuration conf) throws IOException { > this(null, conf, null); > } > > On Mon, Sep 22, 2014 at 7:41 PM, tobe wrote: > > > @qiang I have read about this issue. Is it > > https://issues.apache.org/jira/browse/HBASE-4495? > > > > I look deep in to code and can't find the reason of this. Any suggestion > is > > welcome. > > > > On Mon, Sep 22, 2014 at 6:22 PM, Qiang Tian wrote: > > > > > it looks coming from > > > > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.94.19/org/apache/hadoop/hbase/catalog/CatalogTracker.java#186 > > > > > > if no zookeeper watcher passed in, we will connect to zk...this logic > > > mostly used in client side(getCatalogTracker), so it looks some server > > side > > > code use it as well.. I recall Mikhail has a jira to remove catalog > > > tracker(forgot jira number...) > > > > > > > > > > > > On Sat, Sep 20, 2014 at 7:38 PM, tobe wrote: > > > > > > > Here's the detailed log and you can see it establishes a few sessions > > for > > > > every seconds. It doesn't happen in other clusters but you can see > the > > > > similar log in https://issues.apache.org/jira/browse/HBASE-9779. The > > > HBase > > > > version is 0.9.3 so it print ReplicationSink status as INFO and the > > later > > > > version will set it as DEBUG. > > > > > > > > 2014-09-20,17:34:50,953 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:50,954 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:50,954 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:50,954 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master02.bj/10.101.10.67:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:50,954 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master02.bj/10.101.10.67:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:50,956 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master02.bj/10.101.10.67:11000 > > , > > > > sessionid = 0x14877bf55307a85, negotiated timeout = 30000 > > > > 2014-09-20,17:34:50,962 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x14877bf55307a85 closed > > > > 2014-09-20,17:34:50,962 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:50,963 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 9 > > > > 2014-09-20,17:34:50,984 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:50,985 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:50,985 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:50,985 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master02.bj/10.101.10.67:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:50,985 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master02.bj/10.101.10.67:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:50,987 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master02.bj/10.101.10.67:11000 > > , > > > > sessionid = 0x14877bf55307a87, negotiated timeout = 30000 > > > > 2014-09-20,17:34:50,993 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x14877bf55307a87 closed > > > > 2014-09-20,17:34:50,993 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:50,995 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:50,995 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:50,995 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:50,995 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master02.bj/10.101.10.67:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:50,996 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master02.bj/10.101.10.67:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:50,997 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master02.bj/10.101.10.67:11000 > > , > > > > sessionid = 0x14877bf55307a88, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,003 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x14877bf55307a88 closed > > > > 2014-09-20,17:34:51,003 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,004 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,004 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,004 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,005 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct03.bj/10.2.201.75:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,005 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct03.bj/10.2.201.75:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,006 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > > , > > > > sessionid = 0x44877d1f0dfc467, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,011 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x44877d1f0dfc467 closed > > > > 2014-09-20,17:34:51,011 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,013 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 16 > > > > 2014-09-20,17:34:51,048 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,048 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,048 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,048 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct01.bj/10.2.201.73:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,049 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct01.bj/10.2.201.73:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,050 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > > , > > > > sessionid = 0x24878270100902a, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,056 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x24878270100902a closed > > > > 2014-09-20,17:34:51,056 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,058 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,058 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,058 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,058 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master02.bj/10.101.10.67:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,059 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master02.bj/10.101.10.67:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:51,060 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master02.bj/10.101.10.67:11000 > > , > > > > sessionid = 0x14877bf55307a8b, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,066 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x14877bf55307a8b closed > > > > 2014-09-20,17:34:51,066 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,068 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,068 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,068 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,068 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct02.bj/10.2.201.74:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,069 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct02.bj/10.2.201.74:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,070 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > > , > > > > sessionid = 0x34877bf55f10496, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,075 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x34877bf55f10496 closed > > > > 2014-09-20,17:34:51,075 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,076 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 12 > > > > 2014-09-20,17:34:51,085 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,086 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,086 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,086 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct03.bj/10.2.201.75:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,086 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct03.bj/10.2.201.75:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,087 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > > , > > > > sessionid = 0x44877d1f0dfc469, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,093 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x44877d1f0dfc469 closed > > > > 2014-09-20,17:34:51,093 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,095 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,095 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,095 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,095 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct02.bj/10.2.201.74:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,096 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct02.bj/10.2.201.74:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,097 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > > , > > > > sessionid = 0x34877bf55f10497, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,103 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x34877bf55f10497 closed > > > > 2014-09-20,17:34:51,103 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,105 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,105 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,105 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,106 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct02.bj/10.2.201.74:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,106 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct02.bj/10.2.201.74:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,107 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > > , > > > > sessionid = 0x34877bf55f10498, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,112 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x34877bf55f10498 closed > > > > 2014-09-20,17:34:51,112 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,115 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 14 > > > > 2014-09-20,17:34:51,168 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,168 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,168 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,168 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct03.bj/10.2.201.75:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,169 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct03.bj/10.2.201.75:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,170 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > > , > > > > sessionid = 0x44877d1f0dfc46d, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,175 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,175 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x44877d1f0dfc46d closed > > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,177 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,177 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct03.bj/10.2.201.75:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct03.bj/10.2.201.75:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,179 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > > , > > > > sessionid = 0x44877d1f0dfc46e, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,184 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x44877d1f0dfc46e closed > > > > 2014-09-20,17:34:51,184 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,185 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 2 > > > > 2014-09-20,17:34:51,203 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,204 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,204 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,204 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct02.bj/10.2.201.74:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,204 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct02.bj/10.2.201.74:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,205 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > > , > > > > sessionid = 0x34877bf55f1049c, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,211 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x34877bf55f1049c closed > > > > 2014-09-20,17:34:51,211 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,213 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,213 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,213 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,213 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct01.bj/10.2.201.73:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,214 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct01.bj/10.2.201.73:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,215 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > > , > > > > sessionid = 0x24878270100902c, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,221 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x24878270100902c closed > > > > 2014-09-20,17:34:51,221 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,224 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,224 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,224 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,224 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-hadoop-srv-ct03.bj/10.2.201.75:11000. > > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,225 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-hadoop-srv-ct03.bj/10.2.201.75:11000, > > > > initiating session > > > > 2014-09-20,17:34:51,226 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > > , > > > > sessionid = 0x44877d1f0dfc46f, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,232 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x44877d1f0dfc46f closed > > > > 2014-09-20,17:34:51,232 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,233 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 16 > > > > 2014-09-20,17:34:51,362 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,362 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,362 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,363 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master01.bj/10.101.10.66:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,363 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master01.bj/10.101.10.66:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:51,364 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master01.bj/10.101.10.66:11000 > > , > > > > sessionid = 0x4877f561587a09, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,370 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x4877f561587a09 closed > > > > 2014-09-20,17:34:51,370 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,371 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,371 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,372 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,372 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master01.bj/10.101.10.66:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,372 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master01.bj/10.101.10.66:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:51,373 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master01.bj/10.101.10.66:11000 > > , > > > > sessionid = 0x4877f561587a0b, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,380 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x4877f561587a0b closed > > > > 2014-09-20,17:34:51,380 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,382 INFO org.apache.zookeeper.ZooKeeper: > Initiating > > > > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000 > , > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > sessionTimeout=30000 > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > > > 2014-09-20,17:34:51,382 INFO > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > identifier > > of > > > > this process is 4926@lg-hadoop-srv-st02.bj > > > > 2014-09-20,17:34:51,382 INFO > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > GSSAPI > > > as > > > > SASL mechanism. > > > > 2014-09-20,17:34:51,382 INFO org.apache.zookeeper.ClientCnxn: Opening > > > > socket connection to server lg-com-master01.bj/10.101.10.66:11000. > > Will > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > 2014-09-20,17:34:51,383 INFO org.apache.zookeeper.ClientCnxn: Socket > > > > connection established to lg-com-master01.bj/10.101.10.66:11000, > > > > initiating > > > > session > > > > 2014-09-20,17:34:51,384 INFO org.apache.zookeeper.ClientCnxn: Session > > > > establishment complete on server > lg-com-master01.bj/10.101.10.66:11000 > > , > > > > sessionid = 0x4877f561587a0c, negotiated timeout = 30000 > > > > 2014-09-20,17:34:51,390 INFO org.apache.zookeeper.ZooKeeper: Session: > > > > 0x4877f561587a0c closed > > > > 2014-09-20,17:34:51,390 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > > > shut down > > > > 2014-09-20,17:34:51,391 INFO > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > Total > > > > replicated: 3 > > > > > > > > > > > > On Sat, Sep 20, 2014 at 1:44 PM, lars hofhansl > > wrote: > > > > > > > > > Hi, > > > > > > > > > > can you define "frequently"? > > > > > I.e. send a larger snippet of the log. Connecting every few minutes > > > would > > > > > OK, Multiple times per second would be strange. > > > > > > > > > > -- Lars > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > From: tobe > > > > > To: "user@hbase.apache.org" > > > > > Sent: Thursday, September 18, 2014 1:50 AM > > > > > Subject: HBase establishes session with ZooKeeper and close the > > session > > > > > immediately > > > > > > > > > > > > > > > I have found that our RegionServers connect to the ZooKeeper > > > frequently. > > > > > They seems to constantly establish the session, close it and > > reconnect > > > > the > > > > > ZooKeeper. Here is the log for both server and client sides. I have > > no > > > > idea > > > > > why this happens and how to deal with it? We're using HBase 0.94.11 > > and > > > > > ZooKeeper 3.4.4. > > > > > > > > > > The log from HBase RegionServer: > > > > > > > > > > 2014-09-18,16:38:17,867 INFO org.apache.zookeeper.ZooKeeper: > > Initiating > > > > > client connection, connectString=10.2.201.74:11000, > 10.2.201.73:11000 > > , > > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > > > > sessionTimeout=30000 > > > > > > > > > > > > > > > > > > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@69d892a1 > > > > > 2014-09-18,16:38:17,868 INFO > > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > GSSAPI > > > > as > > > > > SASL mechanism. > > > > > 2014-09-18,16:38:17,868 INFO org.apache.zookeeper.ClientCnxn: > Opening > > > > > socket connection to server > lg-hadoop-srv-ct01.bj/10.2.201.73:11000. > > > > Will > > > > > attempt to SASL-authenticate using Login Context section 'Client' > > > > > 2014-09-18,16:38:17,868 INFO > > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > identifier > > > of > > > > > this process is 11787@lg-hadoop-srv-st05.bj > > > > > 2014-09-18,16:38:17,868 INFO org.apache.zookeeper.ClientCnxn: > Socket > > > > > connection established to lg-hadoop-srv-ct01.bj/10.2.201.73:11000, > > > > > initiating session > > > > > 2014-09-18,16:38:17,870 INFO org.apache.zookeeper.ClientCnxn: > Session > > > > > establishment complete on server > > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > > > , > > > > > sessionid = 0x248782700e52b3c, negotiated timeout = 30000 > > > > > 2014-09-18,16:38:17,876 INFO org.apache.zookeeper.ZooKeeper: > Session: > > > > > 0x248782700e52b3c closed > > > > > 2014-09-18,16:38:17,876 INFO org.apache.zookeeper.ClientCnxn: > > > EventThread > > > > > shut down > > > > > 2014-09-18,16:38:17,878 INFO > > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > Total > > > > > replicated: 24 > > > > > > > > > > The log from its ZooKeeper server: > > > > > > > > > > 2014-09-18,16:38:17,869 INFO > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory: [myid:2] Accepted > > > > socket > > > > > connection from /10.2.201.76:55621 > > > > > 2014-09-18,16:38:17,869 INFO > > > org.apache.zookeeper.server.ZooKeeperServer: > > > > > [myid:2] Client attempting to establish new session at / > > > > 10.2.201.76:55621 > > > > > 2014-09-18,16:38:17,870 INFO > > > org.apache.zookeeper.server.ZooKeeperServer: > > > > > [myid:2] Established session 0x248782700e52b3c with negotiated > > timeout > > > > > 30000 for client /10.2.201.76:55621 > > > > > 2014-09-18,16:38:17,872 INFO > > > > > org.apache.zookeeper.server.auth.SaslServerCallbackHandler: > [myid:2] > > > > > Successfully authenticated client: > > > > > authenticationID=hbase_srv/hadoop@XIAOMI.HADOOP; > > > > > authorizationID=hbase_srv/hadoop@XIAOMI.HADOOP. > > > > > 2014-09-18,16:38:17,872 INFO > > > > > org.apache.zookeeper.server.auth.SaslServerCallbackHandler: > [myid:2] > > > > > Setting authorizedID: hbase_srv > > > > > 2014-09-18,16:38:17,872 INFO > > > org.apache.zookeeper.server.ZooKeeperServer: > > > > > [myid:2] adding SASL authorization for authorizationID: hbase_srv > > > > > 2014-09-18,16:38:17,877 INFO > > org.apache.zookeeper.server.NIOServerCnxn: > > > > > [myid:2] Closed socket connection for client /10.2.201.76:55621 > > which > > > > had > > > > > sessionid 0x248782700e52b3c > > > > > > > > > > > > > > > --001a1134787a33b08e0503b5ab06--