hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Kurucz (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2504) Double assigned znodes in regionserver
Date Thu, 29 Apr 2010 22:27:53 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862449#action_12862449
] 

Miklos Kurucz commented on HBASE-2504:
--------------------------------------

Right, I did not realize there is actually two different connections.
Then my problem is that the shutdown procedure hangs up before actually reaching the zooKeeperWrapper.close()
line in the regionserver.

regionserver log: 
2010-04-29 16:35:47,379 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread
exiting
2010-04-29 16:36:00,620 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -9142708263997773555
lease expired
2010-04-29 16:36:00,620 INFO org.apache.hadoop.hbase.Leases: regionserver/10.1.3.102:60020.leaseChecker
closing leases
2010-04-29 16:36:00,621 INFO org.apache.hadoop.hbase.Leases: regionserver/10.1.3.102:60020.leaseChecker
closed leases
2010-04-29 22:06:01,264 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Starting
shutdown thread
2010-04-29 22:06:01,421 ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file /hbase/.logs/dell102.cluster,60020,1272540873161/hlog.dat.1272551586893
: java.io.IOException: IOException flush:java.io.IOException: All datanodes 10.1.3.127:50010
are bad. Aborting...
java.io.IOException: IOException flush:java.io.IOException: All datanodes 10.1.3.127:50010
are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3149)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
        at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:90)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.hflush(HLog.java:839)
        at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:768)
2010-04-29 22:06:01,441 ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file /hbase/Test5/compaction.dir/2080282894/8986023427157807963
: java.io.IOException: All datanodes 10.1.3.101:50010 are bad. Aborting...
java.io.IOException: All datanodes 10.1.3.101:50010 are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2593)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2137)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2302)
2010-04-29 22:06:01,637 INFO org.apache.zookeeper.ZooKeeper: Session: 0x284958aa550014 closed
2010-04-29 22:06:02,012 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection
with ZooKeeper

Regionserver hung up, until I manually sent a kill signal to it after some time, and then
hung up again.
But this is not related to zookeeper at all, my mistake.


> Double assigned znodes in regionserver
> --------------------------------------
>
>                 Key: HBASE-2504
>                 URL: https://issues.apache.org/jira/browse/HBASE-2504
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.20.3
>            Reporter: Miklos Kurucz
>
> regionserver log:
> Thu Apr 29 13:34:27 CEST 2010 Starting regionserver on dell102
> ...
> 2010-04-29 13:34:29,656 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection
to server dell147/10.1.3.147:2181
> 2010-04-29 13:34:29,657 INFO org.apache.zookeeper.ClientCnxn: Socket connection established
to dell147/10.1.3.147:2181, initiating session
> 2010-04-29 13:34:29,678 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete
on server dell147/10.1.3.147:2181, sessionid = 0x284958aa550001, negotiated timeout = 60000
> ...
> 2010-04-29 14:13:30,096 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection,
connectString=dell149:2181,dell148:2181,dell147:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@46d895e1
> 2010-04-29 14:13:30,096 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection
to server dell147/10.1.3.147:2181
> 2010-04-29 14:13:30,161 INFO org.apache.zookeeper.ClientCnxn: Socket connection established
to dell147/10.1.3.147:2181, initiating session
> 2010-04-29 14:13:30,194 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete
on server dell147/10.1.3.147:2181, sessionid = 0x284958aa550014, negotiated timeout = 60000
> 2010-04-29 14:13:30,195 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read
ZNode /hbase/root-region-server got 10.1.3.123:60020
> 2010-04-29 14:13:30,226 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT at 10.1.3.123:60020
> 2010-04-29 14:13:30,243 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cached location for .META.,,1 is 10.1.3.125:60020
> 2010-04-29 14:13:30,247 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read
ZNode /hbase/master got 10.1.3.150:60000
> ...
> 2010-04-29 22:06:01,637 INFO org.apache.zookeeper.ZooKeeper: Session: 0x284958aa550014
closed
> 2010-04-29 22:06:02,012 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed
connection with ZooKeeper
> Clearly the reinitializeZooKeeper() method was called for some reason.
> Unfortunately:
> hbase(main):005:0> zk 'get /hbase/rs/1272540873161'
> 10.1.3.102:60020
> cZxid = 0x5f0000002b
> ctime = Thu Apr 29 13:34:33 CEST 2010
> mZxid = 0x5f0000003e
> mtime = Thu Apr 29 13:34:33 CEST 2010
> pZxid = 0x5f0000002b
> cversion = 0
> dataVersion = 1
> aclVersion = 0
> ephemeralOwner = 0x284958aa550001
> dataLength = 16
> numChildren = 0
> The owner of the zookeeper node is the first session which was never closed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message