hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tobe <tobeg3oo...@gmail.com>
Subject Re: HBase establishes session with ZooKeeper and close the session immediately
Date Wed, 24 Sep 2014 11:41:31 GMT
Thanks @qiang and I have found the root cause.

Yes, it's about ReplicationSink and I have found the log,
"2014-09-23,14:58:01,736
DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
Replicating for table online_miliao_recent".

Then I look at the relevant code and found that every time it calls
replicateEntries(), it invokes sharedHBaseAdmin.tableExists(table) as well.

The sharedHBaseAdmin.tableExists() will create a new CatalogTracker object
which is also a ZooKeeper client.

When this method exits, it will cleanup the ZooKeeper client and the
session.

So this log looks reasonable because the Replication is running. But the
tableExists() is a little heavy and I don't think we should invoke it for
each time I replicate enties. I also notice that CatalogTracker is not in
ReplicationSink after 0.94.11 so it's not a problem for the later versions.

It would be great if I have found the jira which removes the CatalogTracker
from ReplicationSink :-)



On Wed, Sep 24, 2014 at 6:46 PM, Qiang Tian <tianq01@gmail.com> wrote:

> yeah...replication uses client API to process the edits..
> see ReplicationSink#replicateEntries.
> perhaps adding a stacktrace here is the best(and fastest) way to find the
> root:
>
> 2014-09-18,16:38:17,867 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=10.2.201.74:11000,10.2.201.73:11000,
> 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> sessionTimeout=30000
>
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@69d892a1
>
>
> On Wed, Sep 24, 2014 at 10:11 AM, tobe <tobeg3oogle@gmail.com> wrote:
>
> > It seems to be related to replication. When I disable the replication
> from
> > the other clusters, the problem doesn't occur again. And I print the
> DEBUG
> > log to find some abnormal result. I don't think we should get the
> response
> > of "value=-2209236057317107828" when scan the META table.
> >
> > 2014-09-23,14:57:59,126 DEBUG
> org.apache.hadoop.hbase.client.ClientScanner:
> > Creating scanner over .META. starting at key 'online_miliao_message,,'
> > 2014-09-23,14:57:59,126 DEBUG
> org.apache.hadoop.hbase.client.ClientScanner:
> > Advancing internal scanner to startKey at 'online_miliao_message,,'
> > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.HBaseClient: IPC
> Client
> > (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 from
> > hbase_srv/hadoop@XIAOMI.HADOOP sending #208
> > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureClient: IPC
> > Client (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600
> > from hbase_srv/hadoop@XIAOMI.HADOOP got value #208
> > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureClient: call
> #208
> > state is 0
> > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureClient: call
> > #208, response is:
> > OW[class=long,value=-2209236057317107828]
> > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureRpcEngine:
> Call:
> > openScanner 0
> > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.HBaseClient: IPC
> Client
> > (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 from
> > hbase_srv/hadoop@XIAOMI.HADOOP sending #209
> > 2014-09-23,14:57:59,128 DEBUG
> > org.apache.hadoop.hbase.security.HBaseSaslRpcServer: SASL server GSSAPI
> > callback: setting canonicalized client ID: hbase_srv/hadoop@XIAOMI.HADOOP
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.hbase.ipc.SecureServer:
> > SASL server context established. Negotiated QoP is auth
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.hbase.ipc.SecureServer:
> > SASL server successfully authenticated client:
> > hbase_srv/hadoop@XIAOMI.HADOOP (auth:SIMPLE)
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureClient: IPC
> > Client (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600
> > from hbase_srv/hadoop@XIAOMI.HADOOP got value #209
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureClient: call
> #209
> > state is 0
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.hbase.ipc.SecureServer:
> > Successfully authorized
> >
> org.apache.hadoop.hbase.ipc.HRegionInterface-hbase_srv/hadoop@XIAOMI.HADOOP
> > (auth:SIMPLE)
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureClient: call
> > #209, response is:
> > OW[class=class
> >
> >
> [Lorg.apache.hadoop.hbase.client.Result;,value=[Lorg.apache.hadoop.hbase.client.Result;@6204904d
> > ]
> > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureRpcEngine:
> Call:
> > next 2
> > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.HBaseClient: IPC
> Client
> > (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 from
> > hbase_srv/hadoop@XIAOMI.HADOOP sending #210
> > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureClient: IPC
> > Client (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600
> > from hbase_srv/hadoop@XIAOMI.HADOOP got value #210
> > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureClient: call
> #210
> > state is 0
> > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureClient: call
> > #210, response is:
> > OW[class=void,value=null]
> > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureRpcEngine:
> Call:
> > close 0
> >
> > On Tue, Sep 23, 2014 at 2:27 PM, tobe <tobeg3oogle@gmail.com> wrote:
> >
> > > I mean I have no idea why the RegionServer has so many log about
> > > establishing and closing session.
> > >
> > > I don't think we have so many clients within every second. This happens
> > > when I restart the server but not for all HBase clusters.
> > >
> > > On Tue, Sep 23, 2014 at 11:57 AM, Qiang Tian <tianq01@gmail.com>
> wrote:
> > >
> > >> sorry, what reason do you mean?
> > >> I took a quick look at 0.94.21,  it looks only client code paths call
> > >> below
> > >> constructor, so this version looks immune to the problem..
> > >>
> > >>   public CatalogTracker(final Configuration conf) throws IOException {
> > >>     this(null, conf, null);
> > >>   }
> > >>
> > >> On Mon, Sep 22, 2014 at 7:41 PM, tobe <tobeg3oogle@gmail.com> wrote:
> > >>
> > >> > @qiang I have read about this issue. Is it
> > >> > https://issues.apache.org/jira/browse/HBASE-4495?
> > >> >
> > >> > I look deep in to code and can't find the reason of this. Any
> > >> suggestion is
> > >> > welcome.
> > >> >
> > >> > On Mon, Sep 22, 2014 at 6:22 PM, Qiang Tian <tianq01@gmail.com>
> > wrote:
> > >> >
> > >> > > it looks coming from
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.94.19/org/apache/hadoop/hbase/catalog/CatalogTracker.java#186
> > >> > >
> > >> > > if no zookeeper watcher passed in, we will connect to zk...this
> > logic
> > >> > > mostly used in client side(getCatalogTracker), so it looks some
> > server
> > >> > side
> > >> > > code use it as well.. I recall Mikhail has a jira to remove
> catalog
> > >> > > tracker(forgot jira number...)
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Sat, Sep 20, 2014 at 7:38 PM, tobe <tobeg3oogle@gmail.com>
> > wrote:
> > >> > >
> > >> > > > Here's the detailed log and you can see it establishes a few
> > >> sessions
> > >> > for
> > >> > > > every seconds. It doesn't happen in other clusters but you can
> see
> > >> the
> > >> > > > similar log in https://issues.apache.org/jira/browse/HBASE-9779
> .
> > >> The
> > >> > > HBase
> > >> > > > version is 0.9.3 so it print ReplicationSink status as INFO and
> > the
> > >> > later
> > >> > > > version will set it as DEBUG.
> > >> > > >
> > >> > > > 2014-09-20,17:34:50,953 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:50,954 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:50,954 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:50,954 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master02.bj/10.101.10.67:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:50,954 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:50,956 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master02.bj/10.101.10.67:11000
> > >> > ,
> > >> > > > sessionid = 0x14877bf55307a85, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:50,962 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x14877bf55307a85 closed
> > >> > > > 2014-09-20,17:34:50,962 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:50,963 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 9
> > >> > > > 2014-09-20,17:34:50,984 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:50,985 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:50,985 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:50,985 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master02.bj/10.101.10.67:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:50,985 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:50,987 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master02.bj/10.101.10.67:11000
> > >> > ,
> > >> > > > sessionid = 0x14877bf55307a87, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:50,993 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x14877bf55307a87 closed
> > >> > > > 2014-09-20,17:34:50,993 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:50,995 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:50,995 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:50,995 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:50,995 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master02.bj/10.101.10.67:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:50,996 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:50,997 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master02.bj/10.101.10.67:11000
> > >> > ,
> > >> > > > sessionid = 0x14877bf55307a88, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,003 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x14877bf55307a88 closed
> > >> > > > 2014-09-20,17:34:51,003 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,004 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,004 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,004 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,005 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,005 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,006 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> > > ,
> > >> > > > sessionid = 0x44877d1f0dfc467, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,011 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x44877d1f0dfc467 closed
> > >> > > > 2014-09-20,17:34:51,011 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,013 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 16
> > >> > > > 2014-09-20,17:34:51,048 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,048 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,048 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,048 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,049 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,050 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > >> > > ,
> > >> > > > sessionid = 0x24878270100902a, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,056 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x24878270100902a closed
> > >> > > > 2014-09-20,17:34:51,056 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,058 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,058 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,058 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,058 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master02.bj/10.101.10.67:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,059 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:51,060 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master02.bj/10.101.10.67:11000
> > >> > ,
> > >> > > > sessionid = 0x14877bf55307a8b, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,066 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x14877bf55307a8b closed
> > >> > > > 2014-09-20,17:34:51,066 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,068 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,068 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,068 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,068 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,069 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,070 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> > > ,
> > >> > > > sessionid = 0x34877bf55f10496, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,075 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x34877bf55f10496 closed
> > >> > > > 2014-09-20,17:34:51,075 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,076 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 12
> > >> > > > 2014-09-20,17:34:51,085 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,086 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,086 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,086 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,086 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,087 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> > > ,
> > >> > > > sessionid = 0x44877d1f0dfc469, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,093 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x44877d1f0dfc469 closed
> > >> > > > 2014-09-20,17:34:51,093 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,095 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,095 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,095 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,095 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,096 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,097 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> > > ,
> > >> > > > sessionid = 0x34877bf55f10497, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,103 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x34877bf55f10497 closed
> > >> > > > 2014-09-20,17:34:51,103 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,105 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,105 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,105 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,106 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,106 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,107 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> > > ,
> > >> > > > sessionid = 0x34877bf55f10498, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,112 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x34877bf55f10498 closed
> > >> > > > 2014-09-20,17:34:51,112 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,115 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 14
> > >> > > > 2014-09-20,17:34:51,168 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,168 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,168 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,168 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,169 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,170 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> > > ,
> > >> > > > sessionid = 0x44877d1f0dfc46d, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,175 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,175 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x44877d1f0dfc46d closed
> > >> > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,177 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,177 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,179 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> > > ,
> > >> > > > sessionid = 0x44877d1f0dfc46e, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,184 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x44877d1f0dfc46e closed
> > >> > > > 2014-09-20,17:34:51,184 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,185 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 2
> > >> > > > 2014-09-20,17:34:51,203 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,204 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,204 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,204 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,204 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,205 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000
> > >> > > ,
> > >> > > > sessionid = 0x34877bf55f1049c, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,211 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x34877bf55f1049c closed
> > >> > > > 2014-09-20,17:34:51,211 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,213 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,213 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,213 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,213 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,214 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,215 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > >> > > ,
> > >> > > > sessionid = 0x24878270100902c, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,221 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x24878270100902c closed
> > >> > > > 2014-09-20,17:34:51,221 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,224 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,224 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,224 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,224 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> .
> > >> > > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,225 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to
> lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > ,
> > >> > > > initiating session
> > >> > > > 2014-09-20,17:34:51,226 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000
> > >> > > ,
> > >> > > > sessionid = 0x44877d1f0dfc46f, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,232 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x44877d1f0dfc46f closed
> > >> > > > 2014-09-20,17:34:51,232 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,233 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 16
> > >> > > > 2014-09-20,17:34:51,362 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,362 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,362 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,363 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master01.bj/10.101.10.66:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,363 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master01.bj/10.101.10.66:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:51,364 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master01.bj/10.101.10.66:11000
> > >> > ,
> > >> > > > sessionid = 0x4877f561587a09, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,370 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x4877f561587a09 closed
> > >> > > > 2014-09-20,17:34:51,370 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,371 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,371 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,372 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,372 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master01.bj/10.101.10.66:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,372 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master01.bj/10.101.10.66:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:51,373 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master01.bj/10.101.10.66:11000
> > >> > ,
> > >> > > > sessionid = 0x4877f561587a0b, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,380 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x4877f561587a0b closed
> > >> > > > 2014-09-20,17:34:51,380 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,382 INFO org.apache.zookeeper.ZooKeeper:
> > >> Initiating
> > >> > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000,
> > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > sessionTimeout=30000
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e
> > >> > > > 2014-09-20,17:34:51,382 INFO
> > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> identifier
> > >> > of
> > >> > > > this process is 4926@lg-hadoop-srv-st02.bj
> > >> > > > 2014-09-20,17:34:51,382 INFO
> > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use
> > >> GSSAPI
> > >> > > as
> > >> > > > SASL mechanism.
> > >> > > > 2014-09-20,17:34:51,382 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > socket connection to server
> lg-com-master01.bj/10.101.10.66:11000
> > .
> > >> > Will
> > >> > > > attempt to SASL-authenticate using Login Context section
> 'Client'
> > >> > > > 2014-09-20,17:34:51,383 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > >> > > > connection established to lg-com-master01.bj/10.101.10.66:11000
> ,
> > >> > > > initiating
> > >> > > > session
> > >> > > > 2014-09-20,17:34:51,384 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > establishment complete on server
> > >> lg-com-master01.bj/10.101.10.66:11000
> > >> > ,
> > >> > > > sessionid = 0x4877f561587a0c, negotiated timeout = 30000
> > >> > > > 2014-09-20,17:34:51,390 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > 0x4877f561587a0c closed
> > >> > > > 2014-09-20,17:34:51,390 INFO org.apache.zookeeper.ClientCnxn:
> > >> > EventThread
> > >> > > > shut down
> > >> > > > 2014-09-20,17:34:51,391 INFO
> > >> > > >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> Total
> > >> > > > replicated: 3
> > >> > > >
> > >> > > >
> > >> > > > On Sat, Sep 20, 2014 at 1:44 PM, lars hofhansl <
> larsh@apache.org>
> > >> > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > can you define "frequently"?
> > >> > > > > I.e. send a larger snippet of the log. Connecting every few
> > >> minutes
> > >> > > would
> > >> > > > > OK, Multiple times per second would be strange.
> > >> > > > >
> > >> > > > > -- Lars
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > ________________________________
> > >> > > > >  From: tobe <tobeg3oogle@gmail.com>
> > >> > > > > To: "user@hbase.apache.org" <user@hbase.apache.org>
> > >> > > > > Sent: Thursday, September 18, 2014 1:50 AM
> > >> > > > > Subject: HBase establishes session with ZooKeeper and close
> the
> > >> > session
> > >> > > > > immediately
> > >> > > > >
> > >> > > > >
> > >> > > > > I have found that our RegionServers connect to the ZooKeeper
> > >> > > frequently.
> > >> > > > > They seems to constantly establish the session, close it and
> > >> > reconnect
> > >> > > > the
> > >> > > > > ZooKeeper. Here is the log for both server and client sides. I
> > >> have
> > >> > no
> > >> > > > idea
> > >> > > > > why this happens and how to deal with it? We're using HBase
> > >> 0.94.11
> > >> > and
> > >> > > > > ZooKeeper 3.4.4.
> > >> > > > >
> > >> > > > > The log from HBase RegionServer:
> > >> > > > >
> > >> > > > > 2014-09-18,16:38:17,867 INFO org.apache.zookeeper.ZooKeeper:
> > >> > Initiating
> > >> > > > > client connection, connectString=10.2.201.74:11000,
> > >> 10.2.201.73:11000
> > >> > ,
> > >> > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000
> > >> > > > > sessionTimeout=30000
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@69d892a1
> > >> > > > > 2014-09-18,16:38:17,868 INFO
> > >> > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will
> use
> > >> > GSSAPI
> > >> > > > as
> > >> > > > > SASL mechanism.
> > >> > > > > 2014-09-18,16:38:17,868 INFO org.apache.zookeeper.ClientCnxn:
> > >> Opening
> > >> > > > > socket connection to server
> > >> lg-hadoop-srv-ct01.bj/10.2.201.73:11000.
> > >> > > > Will
> > >> > > > > attempt to SASL-authenticate using Login Context section
> > 'Client'
> > >> > > > > 2014-09-18,16:38:17,868 INFO
> > >> > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> > >> > identifier
> > >> > > of
> > >> > > > > this process is 11787@lg-hadoop-srv-st05.bj
> > >> > > > > 2014-09-18,16:38:17,868 INFO org.apache.zookeeper.ClientCnxn:
> > >> Socket
> > >> > > > > connection established to
> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > >> ,
> > >> > > > > initiating session
> > >> > > > > 2014-09-18,16:38:17,870 INFO org.apache.zookeeper.ClientCnxn:
> > >> Session
> > >> > > > > establishment complete on server
> > >> > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000
> > >> > > > ,
> > >> > > > > sessionid = 0x248782700e52b3c, negotiated timeout = 30000
> > >> > > > > 2014-09-18,16:38:17,876 INFO org.apache.zookeeper.ZooKeeper:
> > >> Session:
> > >> > > > > 0x248782700e52b3c closed
> > >> > > > > 2014-09-18,16:38:17,876 INFO org.apache.zookeeper.ClientCnxn:
> > >> > > EventThread
> > >> > > > > shut down
> > >> > > > > 2014-09-18,16:38:17,878 INFO
> > >> > > > >
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
> > >> > Total
> > >> > > > > replicated: 24
> > >> > > > >
> > >> > > > > The log from its ZooKeeper server:
> > >> > > > >
> > >> > > > > 2014-09-18,16:38:17,869 INFO
> > >> > > > > org.apache.zookeeper.server.NIOServerCnxnFactory: [myid:2]
> > >> Accepted
> > >> > > > socket
> > >> > > > > connection from /10.2.201.76:55621
> > >> > > > > 2014-09-18,16:38:17,869 INFO
> > >> > > org.apache.zookeeper.server.ZooKeeperServer:
> > >> > > > > [myid:2] Client attempting to establish new session at /
> > >> > > > 10.2.201.76:55621
> > >> > > > > 2014-09-18,16:38:17,870 INFO
> > >> > > org.apache.zookeeper.server.ZooKeeperServer:
> > >> > > > > [myid:2] Established session 0x248782700e52b3c with negotiated
> > >> > timeout
> > >> > > > > 30000 for client /10.2.201.76:55621
> > >> > > > > 2014-09-18,16:38:17,872 INFO
> > >> > > > > org.apache.zookeeper.server.auth.SaslServerCallbackHandler:
> > >> [myid:2]
> > >> > > > > Successfully authenticated client:
> > >> > > > > authenticationID=hbase_srv/hadoop@XIAOMI.HADOOP;
> > >> > > > > authorizationID=hbase_srv/hadoop@XIAOMI.HADOOP.
> > >> > > > > 2014-09-18,16:38:17,872 INFO
> > >> > > > > org.apache.zookeeper.server.auth.SaslServerCallbackHandler:
> > >> [myid:2]
> > >> > > > > Setting authorizedID: hbase_srv
> > >> > > > > 2014-09-18,16:38:17,872 INFO
> > >> > > org.apache.zookeeper.server.ZooKeeperServer:
> > >> > > > > [myid:2] adding SASL authorization for authorizationID:
> > hbase_srv
> > >> > > > > 2014-09-18,16:38:17,877 INFO
> > >> > org.apache.zookeeper.server.NIOServerCnxn:
> > >> > > > > [myid:2] Closed socket connection for client /
> 10.2.201.76:55621
> > >> > which
> > >> > > > had
> > >> > > > > sessionid 0x248782700e52b3c
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message