hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anil gupta <anilgupt...@gmail.com>
Subject Re: flume+hbase+zookeeper, can't hang on to zookeeper session
Date Wed, 10 Oct 2012 01:51:59 GMT
My 2 cents:

In past when i have used OpenJDK, i ran into a DNS resolution trouble with
Hadoop/HBase. The problem was resolved when i started using Sun Jdk. I
would suggest you to try the preferred version of Sun Java 1.6. You can
find this info on HBase website. It's worth a try although i am not sure
that this is the problem.

HTH,
Anil Gupta

On Mon, Oct 8, 2012 at 3:56 PM, iain wright <iainwrig@gmail.com> wrote:

> Below is a debug level log of flume trying to connect. Could we
> experiencing this bug:
> https://issues.apache.org/jira/browse/ZOOKEEPER-1550
>
> We are using openjdk version "1.7.0_02"
> OpenJDK Runtime Environment (build 1.7.0_02-b13)
> OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
>
> debug log:
>
> 2012-10-08 15:28:01,324 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:01,325 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:01,326 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:01,327 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:01,328 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:03,322 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:03,323 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:03,332 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:03,334 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,333 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,334 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,334 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
> exception was thrown while closing send thread for session
> 0x138a0620045bf8f : Client session timed out, have not heard from server in
> 4009ms for sessionid 0x138a0620045bf8f
>
> 2012-10-08 15:28:05,410 (conf-file-poller-0) [DEBUG -
>
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
> Checking file:conf/brian.properties for changes
>
> 2012-10-08 15:28:05,435 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
> Disconnecting client for session: 0x138a0620045bf8f
>
> 2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bf8f closed
>
> 2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
> ZooKeeper#close completed in 4123621708ns
>
> 2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection,
> connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
>
> 2012-10-08 15:28:05,438 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.getRootRegion(HBaseClient.java:2512)]
> Finding the -ROOT- region in ZooKeeper
>
> 2012-10-08 15:28:05,438 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
>
> 2012-10-08 15:28:05,439 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2314)] Done
> handling ZooKeeper event: WatchedEvent state:SyncConnected type:None
> path:null
>
> 2012-10-08 15:28:05,439 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
> Ignore any DEBUG exception from ZooKeeper
>
> 2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> 2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-EventThread) [ERROR -
>
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
>
> 2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
> 0x0
>
> 2012-10-08 15:28:05,442 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
> for session: 0x0
>
> 2012-10-08 15:28:05,442 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
>
> 2012-10-08 15:28:05,443 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
> Session establishment request sent on
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
>
> 2012-10-08 15:28:05,444 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,445 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,455 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
> 0x138a0620045bf90, negotiated timeout = 6000
>
> 2012-10-08 15:28:05,456 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:05,456 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,457 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,458 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,459 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:07,450 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:07,452 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:07,454 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:07,455 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,456 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:09,458 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,460 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:09,462 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,462 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
> exception was thrown while closing send thread for session
> 0x138a0620045bf90 : Client session timed out, have not heard from server in
> 4003ms for sessionid 0x138a0620045bf90
>
> 2012-10-08 15:28:09,564 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
> Disconnecting client for session: 0x138a0620045bf90
>
> 2012-10-08 15:28:09,566 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bf90 closed
>
> 2012-10-08 15:28:09,566 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
> ZooKeeper#close completed in 4126386193ns
>
> 2012-10-08 15:28:09,567 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection,
> connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
>
> 2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
> Ignore any DEBUG exception from ZooKeeper
>
> 2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
>
> 2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
> 0x0
>
> 2012-10-08 15:28:09,575 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
> for session: 0x0
>
> 2012-10-08 15:28:09,576 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> 2012-10-08 15:28:09,579 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
>
> 2012-10-08 15:28:09,580 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
> Session establishment request sent on
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
>
> 2012-10-08 15:28:09,583 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,599 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
> 0x138a0620045bf91, negotiated timeout = 6000
>
> 2012-10-08 15:28:09,601 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:09,602 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,603 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:11,588 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:11,591 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,592 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,596 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,604 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,606 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
> exception was thrown while closing send thread for session
> 0x138a0620045bf91 : Client session timed out, have not heard from server in
> 4003ms for sessionid 0x138a0620045bf91
>
> 2012-10-08 15:28:13,708 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
> Disconnecting client for session: 0x138a0620045bf91
>
> 2012-10-08 15:28:13,710 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bf91 closed
>
> 2012-10-08 15:28:13,711 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
> ZooKeeper#close completed in 4137007581ns
>
> 2012-10-08 15:28:13,712 (lifecycleSupervisor-1-0-EventThread) [WARN -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> type:None path:null
>
> 2012-10-08 15:28:13,713 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection,
> connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
>
> 2012-10-08 15:28:13,721 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> EventThread shut down
>
> 2012-10-08 15:28:13,721 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.getRootRegion(HBaseClient.java:2512)]
> Finding the -ROOT- region in ZooKeeper
>
> 2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> 2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2314)] Done
> handling ZooKeeper event: WatchedEvent state:SyncConnected type:None
> path:null
>
> 2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
> Ignore any DEBUG exception from ZooKeeper
>
> 2012-10-08 15:28:13,727 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
>
> 2012-10-08 15:28:13,730 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
>
> 2012-10-08 15:28:13,730 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
> 0x0
>
> 2012-10-08 15:28:13,733 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
> Session establishment request sent on
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
>
> 2012-10-08 15:28:13,734 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
> for session: 0x0
>
> 2012-10-08 15:28:13,761 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:13,772 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
> 0x138a0620045bf92, negotiated timeout = 6000
>
> 2012-10-08 15:28:13,773 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:13,776 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:13,779 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,783 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0  request:: '/hbase/root-region-server,T  response::
> until SASL authentication completes.
> 2012-10-08 15:28:13,786 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null
> response:: nulluntil SASL authentication completes.
> --
> Iain Wright
> Cell: (562) 852-5916
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
> On Mon, Oct 8, 2012 at 2:41 PM, iain wright <iainwrig@gmail.com> wrote:
>
> > We're having some trouble with the Flume & the HbaseSink working. Seems
> we
> > cannot hang on to zookeeper sessions. The flume list recommended I cross
> > post here as this seems to be something up with the hbaseclient &/or
> > zookeeper. In all honesty I am probably just doing something dumb and not
> > seeing it.
> >
> > Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and
> > are in normal prod use. I don't think we are hitting our
> > hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions
> > in ZK DUMP and I believe it defaults to 2k. I can establish new sessions
> > using the CLI from the same server and idle there for 10 minutes without
> > getting dropped. When I start up flume I see the session show up in the
> ZK
> > DUMP, see active TCP connection on the port etc., it just keeps
> > dropping/reconnecting.
> >
> > Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase
> > directories from one of our datanode+regionserver's.
> >
> > *Searched the flume list, this user appeared to have the same problem,
> > not sure how he fixed it though:*
> >
> >
> http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%3CCADGpRhcgztwhuUJi1izMRX2UOKeAWwhYOkKSKkHYmKopAkwHAQ@mail.gmail.com%3E
> >
> > *startup cmd
> > */app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf
> > -f conf/brian.properties  -Dflume.root.logger=INFO,console*
> > *
> > *flume-env.sh*
> > $ cat conf/flume-env.sh
> >
> > # Licensed to the Apache Software Foundation (ASF) under one
> > # or more contributor license agreements.  See the NOTICE file
> > # distributed with this work for additional information
> > # regarding copyright ownership.  The ASF licenses this file
> > # to you under the Apache License, Version 2.0 (the
> > # "License"); you may not use this file except in compliance
> > # with the License.  You may obtain a copy of the License at
> > #
> > #     http://www.apache.org/licenses/LICENSE-2.0
> > #
> > # Unless required by applicable law or agreed to in writing, software
> > # distributed under the License is distributed on an "AS IS" BASIS,
> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> > # See the License for the specific language governing permissions and
> > # limitations under the License.
> >
> > # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be
> sourced
> > # during Flume startup.
> >
> > # Enviroment variables can be set here.
> >
> > #JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> > # Give Flume more memory and pre-allocate, enable remote monitoring via
> JMX
> > #JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
> >
> > # Note that the Flume conf directory is always included in the classpath.
> > #FLUME_CLASSPATH="/app/flume/lib"
> >
> > FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"
> >
> > HBASE_HOME="/app/hbase-0.94.0"
> >
> > HADOOP_HOME="/app/hadoop-1.0.1"
> >
> >
> > *config*
> > $ cat conf/brian.properties
> > #example.conf: A single-node Flume configuration
> >
> > # Name the components on this agent
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = exec
> > agent1.sources.source1.command = tail -F /tank/log_dev.log
> > agent1.sources.source1.batchSize = 1
> > # Describe sink1
> > #agent1.sinks.sink1.type = logger
> > agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> > agent1.sinks.sink1.table = brian_test
> > agent1.sinks.sink1.columnFamily = f1
> > #agent1.sinks.sink1.serializer =
> > org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
> > #agent1.sinks.sink1.serializer =
> > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> >
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > *flume console log*
> > 2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> >
> environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.io.tmpdir=/var/tmp/
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.compiler=<NA>
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.name=FreeBSD
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.arch=amd64
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.version=9.0-RELEASE
> > 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.name=czhang
> > 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.home=/users/czhang
> > 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
> > 2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection, connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000
> watcher=hconnection
> > 2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)]
> > The identifier of this process is 12494@app14.app.domain.com
> > 2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:06:19,049 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:06:19,072 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:08:19,070 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120001ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:08:19,233 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:08:19,234 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> > Sleeping 2000ms before retry #1...
> > 2012-10-08 12:08:20,691 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:08:20,692 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:08:20,695 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:10:20,695 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120000ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> > Sleeping 4000ms before retry #2...
> > 2012-10-08 12:10:22,792 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:10:22,794 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:10:22,799 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:12:22,800 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120001ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:12:22,902 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:12:22,903 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> > Sleeping 8000ms before retry #3...
> > 2012-10-08 12:12:24,289 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:12:24,291 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:12:24,294 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:14:24,294 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120000ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [ERROR -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:197)]
> > ZooKeeper exists failed after 3 retries
> > 2012-10-08 12:14:24,398 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:239)]
> >
> hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
> > Unable to set watcher on znode /hbase/master
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >         at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >         at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> >         at
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
> >         at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
> >         at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> >         at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
> >         at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> >         at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >         at
> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> >         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> >         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> >         at
> org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
> >         at
> >
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> >         at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> >         at
> >
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >         at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> >         at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >         at java.lang.Thread.run(Thread.java:722)
> > 2012-10-08 12:14:24,405 (lifecycleSupervisor-1-1) [ERROR -
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.keeperException(ZooKeeperWatcher.java:408)]
> >
> hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
> > Received unexpected KeeperException, re-throwing exception
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >         at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >         at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> >         at
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
> >         at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
> >         at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> >         at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
> >         at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> >         at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >         at
> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> >         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> >         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> >         at
> org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
> >         at
> >
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> >         at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> >         at
> >
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >         at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> >         at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >         at java.lang.Thread.run(Thread.java:722)
> > 2012-10-08 12:14:24,407 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1654)]
> > This client just lost it's session with ZooKeeper, will automatically
> > reconnect when needed.
> > 2012-10-08 12:14:26,166 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:14:26,167 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:14:26,170 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> >
> > *zookeeper log*
> > 2012-10-08 19:06:19,049 INFO
> > org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> > connection from /10.10.10.64:16755
> > 2012-10-08 19:06:19,053 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Client attempting to establish new session at /10.10.10.64:16755
> > 2012-10-08 19:06:19,069 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Established session 0x138a0620045bbce with negotiated timeout 180000 for
> > client /10.10.10.64:16755
> > 2012-10-08 19:08:19,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /10.10.10.64:16755 which had
> > sessionid 0x138a0620045bbce
> > 2012-10-08 19:08:20,692 INFO
> > org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> > connection from /10.10.10.64:33245
> > 2012-10-08 19:08:20,693 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Client attempting to renew session 0x138a0620045bbce at /
> 10.10.10.64:33245
> > 2012-10-08 19:08:20,694 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Established session 0x138a0620045bbce with negotiated timeout 180000 for
> > client /10.10.10.64:33245
> > 2012-10-08 19:10:20,695 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > caught end of stream exception
> > EndOfStreamException: Unable to read additional data from client
> sessionid
> > 0x138a0620045bbce, likely client has closed socket
> >     at
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> >     at
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
> >     at java.lang.Thread.run(Thread.java:619)
> > 2012-10-08 19:10:20,698 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /10.10.10.64:33245 which had
> > sessionid 0x138a0620045bbce
> > 2012-10-08 19:10:22,794 INFO
> > org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> > connection from /10.10.10.64:44993
> > 2012-10-08 19:10:22,796 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Client attempting to renew session 0x138a0620045bbce at /
> 10.10.10.64:44993
> > 2012-10-08 19:10:22,798 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Established session 0x138a0620045bbce with negotiated timeout 180000 for
> > client /10.10.10.64:44993
> >
> > *We also tried the hbase async sink as that is recommended over in the
> > flume group, and got similar ** results (but seemingly more serious
> > CONNECTIONLOSS errors)*
> >
> > 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
> > Starting new configuration:{
> > sourceRunners:{source1=EventDrivenSourceRunner: {
> > source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }}
> > sinkRunners:{sink1=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@3968a6fecounterGroup:{
> > name:null counters:{} } }}
> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
> channel1}} }
> > 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
> > Starting Channel channel1
> > 2012-10-08 14:38:29,641 (lifecycleSupervisor-1-0) [INFO -
> >
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
> > Component type: CHANNEL, name: channel1 started
> > 2012-10-08 14:38:29,642 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
> > Starting Sink sink1
> > 2012-10-08 14:38:29,642 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
> > Component type: SINK, name: sink1 started
> > 2012-10-08 14:38:29,643 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
> > Starting Source source1
> > 2012-10-08 14:38:29,643 (lifecycleSupervisor-1-0) [INFO -
> > org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec
> source
> > starting with command:tail -F
> > /tank/clients/telescope/users/czhang/connecttwitter_dev.log
> > 2012-10-08 14:38:29,816 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:zookeeper.version=3.4.4-1386507, built on 09/17/2012 08:33
> GMT
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:host.name=app14.app.domain.com
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.version=1.7.0_02
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.vendor=Oracle Corporation
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.home=/usr/local/openjdk7/jre
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> >
> environment:java.class.path=/app/flume/conf:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-avro-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-log4jappender-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-lang-2.5.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.2.7.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-scribe-source-1.3.0-20120929.012216-17.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-pool-1.5.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-ipc-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-thrift-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpclient-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/mina-core-2.0.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-mapper-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/servlet-api-2.5-20110124.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jcl-over-slf4j-1.7.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jsr305-1.3.9.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.4.0.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-util-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-hdfs-sink-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-collections-3.2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/async-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/paranamer-2.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/velocity-1.7.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-codec-1.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-hbase-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/irclib-1.10.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-jdbc-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-dbcp-1.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-api-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/protobuf-java-2.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-irc-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/joda-time-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-io-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-file-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-recoverable-memory-channel-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-cli-1.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/asynchbase-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/gson-2.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/derby-10.8.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-logging-1.1.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/guava-10.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpcore-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/libthrift-0.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/log4j-1.2.16.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/zookeeper-3.4.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/snappy-java-1.0.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-core-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib:/app/hadoop-1.0.1/libexec/../conf:/usr/local/lib/tools.jar:/app/hadoop-1.0.1/libexec/..:/app/hadoop-1.0.1/libexec/../hadoop-core-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/asm-3.2.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjrt-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjtools-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-1.7.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-cli-1.2.jar:/app/hadoop-1.0.1/libexec/../lib/commons-codec-1.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-collections-3.2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-configuration-1.6.jar:/app/hadoop-1.0.1/libexec/../lib/commons-daemon-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-digester-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/commons-el-1.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-httpclient-3.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-lang-2.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-1.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-api-1.0.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-math-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-net-1.4.1.jar:/app/hadoop-1.0.1/libexec/../lib/core-3.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-capacity-scheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-fairscheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-thriftfs-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hsqldb-1.8.0.10.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-compiler-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-runtime-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jdeb-0.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-core-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-json-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-server-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jets3t-0.6.1.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-util-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jsch-0.1.42.jar:/app/hadoop-1.0.1/libexec/../lib/junit-4.5.jar:/app/hadoop-1.0.1/libexec/../lib/kfs-0.2.2.jar:/app/hadoop-1.0.1/libexec/../lib/log4j-1.2.15.jar:/app/hadoop-1.0.1/libexec/../lib/mockito-all-1.8.5.jar:/app/hadoop-1.0.1/libexec/../lib/oro-2.0.8.jar:/app/hadoop-1.0.1/libexec/../lib/servlet-api-2.5-20081211.jar:/app/hadoop-1.0.1/libexec/../lib/xmlenc-0.52.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/app/hbase-0.94.0/bin/../conf:/usr/local/lib/tools.jar:/app/hbase-0.94.0/bin/..:/app/hbase-0.94.0/bin/../hbase-0.94.0-tests.jar:/app/hbase-0.94.0/bin/../hbase-0.94.0.jar:/app/hbase-0.94.0/bin/../lib/activation-1.1.jar:/app/hbase-0.94.0/bin/../lib/asm-3.1.jar:/app/hbase-0.94.0/bin/../lib/avro-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/avro-ipc-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-1.7.0.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-core-1.8.0.jar:/app/hbase-0.94.0/bin/../lib/commons-cli-1.2.jar:/app/hbase-0.94.0/bin/../lib/commons-codec-1.4.jar:/app/hbase-0.94.0/bin/../lib/commons-collections-3.2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-configuration-1.6.jar:/app/hbase-0.94.0/bin/../lib/commons-digester-1.8.jar:/app/hbase-0.94.0/bin/../lib/commons-el-1.0.jar:/app/hbase-0.94.0/bin/../lib/commons-httpclient-3.1.jar:/app/hbase-0.94.0/bin/../lib/commons-io-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-lang-2.5.jar:/app/hbase-0.94.0/bin/../lib/commons-logging-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/commons-math-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-net-1.4.1.jar:/app/hbase-0.94.0/bin/../lib/core-3.1.1.jar:/app/hbase-0.94.0/bin/../lib/guava-r09.jar:/app/hbase-0.94.0/bin/../lib/hadoop-core-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/high-scale-lib-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/httpclient-4.1.2.jar:/app/hbase-0.94.0/bin/../lib/httpcore-4.1.3.jar:/app/hbase-0.94.0/bin/../lib/jackson-core-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-jaxrs-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-mapper-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-xc-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jamon-runtime-2.3.1.jar:/app/hbase-0.94.0/bin/../lib/jasper-compiler-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jasper-runtime-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jaxb-api-2.1.jar:/app/hbase-0.94.0/bin/../lib/jaxb-impl-2.1.12.jar:/app/hbase-0.94.0/bin/../lib/jersey-core-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-json-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-server-1.4.jar:/app/hbase-0.94.0/bin/../lib/jettison-1.1.jar:/app/hbase-0.94.0/bin/../lib/jetty-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jetty-util-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jruby-complete-1.6.5.jar:/app/hbase-0.94.0/bin/../lib/jsp-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/libthrift-0.8.0.jar:/app/hbase-0.94.0/bin/../lib/log4j-1.2.16.jar:/app/hbase-0.94.0/bin/../lib/netty-3.2.4.Final.jar:/app/hbase-0.94.0/bin/../lib/protobuf-java-2.4.0a.jar:/app/hbase-0.94.0/bin/../lib/servlet-api-2.5-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/snappy-java-1.0.3.2.jar:/app/hbase-0.94.0/bin/../lib/stax-api-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/velocity-1.7.jar:/app/hbase-0.94.0/bin/../lib/xmlenc-0.52.jar:/app/hbase-0.94.0/bin/../lib/zookeeper-3.4.3.jar:.:/app/hbase-0.94.0/conf
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> >
> environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.io.tmpdir=/var/tmp/
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.compiler=<NA>
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.name=FreeBSD
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.arch=amd64
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.version=9.0-RELEASE
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.name=czhang
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.home=/tank/clients/telescope/users/czhang
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
> > 2012-10-08 14:38:29,822 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:29,863 (lifecycleSupervisor-1-1) [INFO -
> >
> org.hbase.async.HBaseClient$ZKClient.getDeferredRoot(HBaseClient.java:2238)]
> > Need to find the -ROOT- region
> > 2012-10-08 14:38:29,870 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:29,879 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:29,908 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdb5, negotiated timeout = 6000
> > 2012-10-08 14:38:33,906 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 4001ms for
> > sessionid 0x138a0620045bdb5, closing socket connection and attempting
> > reconnect
> > 2012-10-08 14:38:34,013 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdb5 closed
> > 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:34,971 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:34,972 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:34,986 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdba, negotiated timeout = 6000
> > 2012-10-08 14:38:39,088 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdba closed
> > 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [WARN -
> > org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> > longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> > type:None path:null
> > 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:39,091 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:39,092 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:39,163 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbb, negotiated timeout = 6000
> > 2012-10-08 14:38:43,164 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 4001ms for
> > sessionid 0x138a0620045bdbb, closing socket connection and attempting
> > reconnect
> > 2012-10-08 14:38:43,265 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:43,683 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdbb closed
> > 2012-10-08 14:38:43,684 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:43,686 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:43,687 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:43,688 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:43,701 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbc, negotiated timeout = 6000
> > 2012-10-08 14:38:47,802 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdbc closed
> > 2012-10-08 14:38:47,804 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:47,807 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:47,810 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:47,812 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:47,821 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbd, negotiated timeout = 6000
> > 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdbd closed
> > 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [WARN -
> > org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> > longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> > type:None path:null
> > 2012-10-08 14:38:51,923 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:51,925 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:51,926 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:51,945 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbe, negotiated timeout = 6000
> >
> > Thanks for the continued help from the community on getting us going
> > w/flume
> >
> > Cheers,
> >
> > --
> > Iain Wright
> > Cell: (562) 852-5916
> >
> > <http://www.labctsi.org/>
> > This email message is confidential, intended only for the recipient(s)
> > named above and may contain information that is privileged, exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> do
> > not disclose or disseminate the message to anyone except the intended
> > recipient. If you have received this message in error, or are not the
> named
> > recipient(s), please immediately notify the sender by return email, and
> > delete all copies of this message.
> >
>



-- 
Thanks & Regards,
Anil Gupta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message