Return-Path: Delivered-To: apmail-hbase-user-archive@www.apache.org Received: (qmail 5667 invoked from network); 27 May 2010 15:08:58 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 May 2010 15:08:58 -0000 Received: (qmail 60014 invoked by uid 500); 27 May 2010 15:08:57 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 59981 invoked by uid 500); 27 May 2010 15:08:57 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 59973 invoked by uid 500); 27 May 2010 15:08:57 -0000 Delivered-To: apmail-hadoop-hbase-user@hadoop.apache.org Received: (qmail 59970 invoked by uid 99); 27 May 2010 15:08:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 15:08:57 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nazario.lucas@gmail.com designates 209.85.160.176 as permitted sender) Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 15:08:48 +0000 Received: by gyf1 with SMTP id 1so34504gyf.35 for ; Thu, 27 May 2010 08:08:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=Uz9E1A/7C6u2BZ1n9Z3+Zuldd4hsHTBn5jR5+itF56Q=; b=SXB4dmxRl8Ez2tdOqxzBgO3qHdTdeSxkhsr0axqcbyLl/O1dEzfCPloDP00FOPEwH/ KT/awqggXsjAtcVt+fr7brabDWvrIrtqQFLbkg9Bw5Le5ONhpIjwTN1X5BvEWD/BpyK1 3UfzWouvWTvzgKkbQRgdIulAD2f5Y+JjRWAp4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Sbz6laYvZaIVZ7Ls99fTO8CC/1yx1Fk5M+d2rrr3PIkabOet74JoWqsjO2v2NyiV5V IlBjtfWdVk0u/eyJ3mXWI/zSWBSLH6OCb+6ra6mvyNLsIFl7UNHKikFUkOnUez/rHFig X1nrbUO2jNtqTpf7BpJ+W/MKePcUyDTl2vVSk= MIME-Version: 1.0 Received: by 10.151.57.13 with SMTP id j13mr284263ybk.236.1274972906758; Thu, 27 May 2010 08:08:26 -0700 (PDT) Received: by 10.150.140.2 with HTTP; Thu, 27 May 2010 08:08:26 -0700 (PDT) Date: Thu, 27 May 2010 12:08:26 -0300 Message-ID: Subject: Zookeeper apparently going down From: =?ISO-8859-1?Q?Lucas_Naz=E1rio_dos_Santos?= To: hbase-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00151750e2ea34a4db048794c315 X-Virus-Checked: Checked by ClamAV on apache.org --00151750e2ea34a4db048794c315 Content-Type: text/plain; charset=ISO-8859-1 Hi, I'm executing HBase 0.20.4 over Hadoop 0.20.2 on Amazon EC2 and it had been running smooth for more than a month. Today it crashed apparently because of some Zookeeper exception. It's a one-machine-only configuration (that will add more nodes in the future) and I'm conscious that it makes the the whole structure more susceptible to errors. Some facts: (1) The HBase master log still reports that my server is up even after the error: 2010-05-27 09:53:40,812 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 75.0 (2) I have plenty of available memory. (3) Time out configurations were relaxed. (3) The region server log contains no errors. Shouldn't Zookeeper recovery nicely? How can I prevent such error from happening in the future? Thanks in advance, Lucas MASTER LOG 2010-05-27 08:40:40,676 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 75.0 2010-05-27 08:40:44,252 ERROR org.apache.hadoop.hbase.master.BaseScanner: Closing scanner java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) at $Proxy1.close(Unknown Source) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:215) at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) 2010-05-27 08:40:44,259 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region java.io.IOException: Call to /10.251.158.224:60020 failed on local exception: java.io.IOException: Connection reset by peer at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:769) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) at $Proxy1.get(Unknown Source) at org.apache.hadoop.hbase.master.BaseScanner.checkAssigned(BaseScanner.java:543) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:192) at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(Unknown Source) at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) at sun.nio.ch.IOUtil.read(Unknown Source) at sun.nio.ch.SocketChannelImpl.read(Unknown Source) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.FilterInputStream.read(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at java.io.DataInputStream.readInt(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:506) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:450) 2010-05-27 08:40:44,253 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x128c8b303040000 to sun.nio.ch.SelectionKeyImpl@470b9279 java.io.IOException: TIMED OUT at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) 2010-05-27 08:40:44,253 ERROR org.apache.hadoop.hbase.master.BaseScanner: Closing scanner java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) at $Proxy1.close(Unknown Source) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:215) at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) 2010-05-27 08:40:44,270 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan one META region: {server: 10.251.158.224:60020, regionname: .META.,,1, startKey: <>} java.io.IOException: Call to /10.251.158.224:60020 failed on local exception: java.io.IOException: Connection reset by peer at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:769) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) at $Proxy1.next(Unknown Source) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:179) at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(Unknown Source) at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) at sun.nio.ch.IOUtil.read(Unknown Source) at sun.nio.ch.SocketChannelImpl.read(Unknown Source) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.FilterInputStream.read(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at java.io.DataInputStream.readInt(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:506) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:450) 2010-05-27 08:40:44,269 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.251.158.224:60020, regionname: -ROOT-,,0, startKey: <>} 2010-05-27 08:40:44,272 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) at $Proxy1.openScanner(Unknown Source) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:177) at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) 2010-05-27 08:40:45,238 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned 2010-05-27 08:40:45,238 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: 10.251.158.224:60020, regionname: .META.,,1, startKey: <>} 2010-05-27 08:40:45,238 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan one META region: {server: 10.251.158.224:60020, regionname: .META.,,1, startKey: <>} java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) at $Proxy1.openScanner(Unknown Source) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:177) at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) 2010-05-27 08:40:45,241 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned 2010-05-27 08:40:45,550 DEBUG org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event None with path null 2010-05-27 08:40:45,711 DEBUG org.apache.hadoop.hbase.master.HMaster: Got event None with path null 2010-05-27 08:40:46,628 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server ip-10-251-158-224.ec2.internal/10.251.158.224:2181 2010-05-27 08:40:46,629 INFO org.apache.zookeeper.ClientCnxn: Priming connection to java.nio.channels.SocketChannel[connected local=/ 10.251.158.224:39413 remote=ip-10-251-158-224.ec2.internal/ 10.251.158.224:2181] 2010-05-27 08:40:46,629 INFO org.apache.zookeeper.ClientCnxn: Server connection successful 2010-05-27 08:40:46,630 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x128c8b303040000 to sun.nio.ch.SelectionKeyImpl@744e022c java.io.IOException: Session Expired at org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) 2010-05-27 08:40:46,631 DEBUG org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event None with path null 2010-05-27 08:40:46,631 DEBUG org.apache.hadoop.hbase.master.HMaster: Got event None with path null 2010-05-27 08:40:46,631 INFO org.apache.hadoop.hbase.master.HMaster: Master lost its znode, trying to get a new one 2010-05-27 08:40:46,631 INFO org.apache.zookeeper.ZooKeeper: Closing session: 0x128c8b303040000 2010-05-27 08:40:46,631 INFO org.apache.zookeeper.ClientCnxn: Closing ClientCnxn for session: 0x128c8b303040000 2010-05-27 08:40:46,651 INFO org.apache.zookeeper.ClientCnxn: Disconnecting ClientCnxn for session: 0x128c8b303040000 2010-05-27 08:40:46,651 INFO org.apache.zookeeper.ZooKeeper: Session: 0x128c8b303040000 closed 2010-05-27 08:40:46,651 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection with ZooKeeper 2010-05-27 08:40:46,684 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-251-158-224.ec2.internal:2181 sessionTimeout=600000 watcher=Thread[HMaster,5,main] 2010-05-27 08:40:46,686 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server ip-10-251-158-224.ec2.internal/10.251.158.224:2181 2010-05-27 08:40:47,097 INFO org.apache.zookeeper.ClientCnxn: Priming connection to java.nio.channels.SocketChannel[connected local=/ 10.251.158.224:39414 remote=ip-10-251-158-224.ec2.internal/ 10.251.158.224:2181] 2010-05-27 08:40:47,097 INFO org.apache.zookeeper.ClientCnxn: Server connection successful 2010-05-27 08:40:47,115 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master 2010-05-27 08:40:47,117 DEBUG org.apache.hadoop.hbase.master.HMaster: Got event None with path null 2010-05-27 08:40:47,121 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Wrote master address 10.251.158.224:60000 to ZooKeeper 2010-05-27 08:40:47,127 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: State node exists. 2010-05-27 08:40:47,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master got 10.251.158.224:60000 2010-05-27 08:40:47,129 DEBUG org.apache.hadoop.hbase.master.HMaster: Checking cluster state... 2010-05-27 08:40:47,130 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.251.158.224:60020 2010-05-27 08:40:47,131 DEBUG org.apache.hadoop.hbase.master.HMaster: This is a fresh start, proceeding with normal startup 2010-05-27 08:40:47,134 INFO org.apache.hadoop.hbase.master.HMaster: Found log folder : ip-10-251-158-224.ec2.internal,60020,1274677721442 2010-05-27 08:40:47,134 INFO org.apache.hadoop.hbase.master.HMaster: Log folder belongs to an existing region server 2010-05-27 08:40:47,134 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2010-05-27 08:41:40,676 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 75.0 2010-05-27 08:41:44,270 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.251.158.224:60020, regionname: -ROOT-,,0, startKey: <>} 2010-05-27 08:41:44,271 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) ZOOKEEPER LOG 2010-05-27 08:40:44,230 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x128c8b303040002 2010-05-27 08:40:44,232 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x128c8b303040001 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/ 10.251.158.224:52406] 2010-05-27 08:40:44,233 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination request for id: 0x128c8b303040582 2010-05-27 08:40:44,236 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination request for id: 0x128c8b303040000 2010-05-27 08:40:44,236 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination request for id: 0x128c8b303040002 2010-05-27 08:40:44,237 WARN org.apache.zookeeper.server.ZooKeeperServer: Ignoring unexpected exception java.io.IOException: Missing session 0x128c8b303040002 at org.apache.zookeeper.server.ZooKeeperServer.touch(ZooKeeperServer.java:337) at org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:570) at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:569) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:411) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:239) 2010-05-27 08:40:44,243 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x128c8b303040002 due to java.io.IOException: Read error 2010-05-27 08:40:44,244 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x128c8b303040002 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/ 10.251.158.224:39520] 2010-05-27 08:40:44,246 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x128c8b303040582 due to java.io.IOException: Read error 2010-05-27 08:40:44,246 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x128c8b303040582 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/ 10.251.158.224:51069] 2010-05-27 08:40:44,247 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.251.158.224:39398 lastZxid 233730 2010-05-27 08:40:44,249 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x128c8b303040002 valid:false 2010-05-27 08:40:44,249 INFO org.apache.zookeeper.server.NIOServerCnxn: Renewing session 0x128c8b303040002 2010-05-27 08:40:44,249 WARN org.apache.zookeeper.server.ZooKeeperServer: Ignoring unexpected exception java.io.IOException: Missing session 0x128c8b303040002 at org.apache.zookeeper.server.ZooKeeperServer.touch(ZooKeeperServer.java:337) at org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:570) at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:569) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:411) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:239) 2010-05-27 08:40:44,253 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x128c8b303040002 due to java.io.IOException: Read error 2010-05-27 08:40:44,254 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x128c8b303040002 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/ 10.251.158.224:39398] 2010-05-27 08:40:45,239 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x128c8b303040000 due to java.io.IOException: Read error 2010-05-27 08:40:45,239 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x128c8b303040000 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/ 10.251.158.224:52405] 2010-05-27 08:40:46,629 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.251.158.224:39413 lastZxid 233729 2010-05-27 08:40:46,630 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x128c8b303040000 valid:false 2010-05-27 08:40:46,630 INFO org.apache.zookeeper.server.NIOServerCnxn: Renewing session 0x128c8b303040000 2010-05-27 08:40:46,631 WARN org.apache.zookeeper.server.ZooKeeperServer: Ignoring unexpected exception java.io.IOException: Missing session 0x128c8b303040000 at org.apache.zookeeper.server.ZooKeeperServer.touch(ZooKeeperServer.java:337) at org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:570) at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:569) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:411) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:239) 2010-05-27 08:40:46,631 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x128c8b303040000 due to java.io.IOException: Read error 2010-05-27 08:40:46,631 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x128c8b303040000 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/ 10.251.158.224:39413] 2010-05-27 08:40:47,098 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.251.158.224:39414 lastZxid 0 2010-05-27 08:40:47,098 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x128c8b303040f9a 2010-05-27 08:40:47,116 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x128c8b303040f9a valid:true 2010-05-27 08:40:47,124 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x128c8b303040f9a type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) 2010-05-27 08:40:56,782 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.251.158.224:39444 lastZxid 0 2010-05-27 08:40:56,782 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x128c8b303040f9b 2010-05-27 08:40:56,858 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x128c8b303040f9b valid:true 2010-05-27 08:41:12,483 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.251.158.224:39463 lastZxid 0 2010-05-27 08:41:12,484 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x128c8b303040f9c 2010-05-27 08:41:12,513 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x128c8b303040f9c valid:true 2010-05-27 08:41:15,396 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.251.158.224:39468 lastZxid 0 2010-05-27 08:41:15,396 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x128c8b303040f9d 2010-05-27 08:41:15,463 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x128c8b303040f9d valid:true --00151750e2ea34a4db048794c315--