hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiang Hua <bea...@gmail.com>
Subject Re: hmaster and regionserver died
Date Tue, 16 Oct 2012 06:47:00 GMT
HI,
   GC log is not opened. only zk's log, pls see below:
2012-10-12 00:14:30,470 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket
connection from /10.20.16.22:56954
2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] -
Unexpected Exception:
java.nio.channels.CancelledKeyException
 at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
 at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
 at
org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545)
 at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1156)
 at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:383)
 at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:267)
 at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2012-10-12 00:14:30,470 - INFO  [QuorumPeer:/0.0.0.0:2181:NIOServerCnxn@1573]
- Invalid session 0x339c539ba64130b for client /10.20.16.29:41178, probably
expired
2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] -
Unexpected Exception:
java.nio.channels.CancelledKeyException
 at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
 at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
 at
org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545)
 at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1156)
 at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:383)
 at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:267)
 at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2012-10-12 00:14:30,470 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770] - Client attempting to renew
session 0x239c539ba632df9 at /10.20.16.23:37706
2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] -
Unexpected Exception:
java.nio.channels.CancelledKeyException


Best R.
beatls

On Mon, Oct 15, 2012 at 4:00 PM, Ramkrishna.S.Vasudevan <
ramkrishna.vasudevan@huawei.com> wrote:

> Hi
>
> Sorry if my reply mislead you.  I  meant to see the GC logs that should
> give
> you an idea of if Full GC happened.
>
> Regards
> Ram
>
> > -----Original Message-----
> > From: Xiang Hua [mailto:beatls@gmail.com]
> > Sent: Monday, October 15, 2012 12:42 PM
> > To: user@hbase.apache.org
> > Subject: Re: hmaster and regionserver died
> >
> > We will check the zk log.
> >
> > On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote:
> >
> > > Check your GC configurations.  Seems to that a Full GC has happened
> > and the
> > > Zookeeper thought that to be session expiry.
> > >
> > > Regards
> > > Ram
> > >
> > > > -----Original Message-----
> > > > From: Xiang Hua [mailto:beatls@gmail.com]
> > > > Sent: Saturday, October 13, 2012 6:20 PM
> > > > To: user@hbase.apache.org
> > > > Subject: hmaster and regionserver died
> > > >
> > > > Hi,
> > > >    the HMaster died as well as regionservers, below is hmaster's
> > log.
> > > > could
> > > > you please find what's problem?
> > > >
> > > >
> > > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > > 10.20.16.34:2181, initiating session
> > > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn:
> > Session
> > > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-
> > stor-
> > > > 3/
> > > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout
> > =
> > > > 40000
> > > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn:
> > Client
> > > > session timed out, have not heard from server in 15046ms for
> > sessionid
> > > > 0x239c539ba630001, closing socket connection and attempting
> > reconnect
> > > > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn:
> > Opening
> > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > > 10.20.16.33:2181
> > > > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn:
> > Client
> > > > session timed out, have not heard from server in 15245ms for
> > sessionid
> > > > 0x139c539bc090003, closing socket connection and attempting
> > reconnect
> > > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn:
> > Opening
> > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > > 10.20.16.33:2181
> > > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > > 10.20.16.33:2181, initiating session
> > > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> > > > EventThread
> > > > shut down
> > > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> > Unable to
> > > > reconnect to ZooKeeper service, session 0x139c539bc090003 has
> > expired,
> > > > closing socket connection
> > > > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > > 10.20.16.33:2181, initiating session
> > > > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn:
> > Session
> > > > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-
> > 2/
> > > > 10.20.16.33:2181, unexpected error, closing socket connection and
> > > > attempting reconnect
> > > > java.io.IOException: Connection reset by peer
> > > >     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> > > >     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> > > >     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> > > >     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> > > >     at
> > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> > > >     at
> > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
> > > >     at
> > > >
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> > > > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn:
> > Opening
> > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > > 10.20.16.34:2181
> > > > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn:
> > Client
> > > > session timed out, have not heard from server in 14007ms for
> > sessionid
> > > > 0x239c539ba630001, closing socket connection and attempting
> > reconnect
> > > > 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn:
> > Opening
> > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
> > > > 10.20.16.32:2181
> > > > 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn:
> > Client
> > > > session timed out, have not heard from server in 26666ms for
> > sessionid
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message