hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zhengbing li <li.zhengb...@huawei.com>
Subject why does regionserver close with exception
Date Fri, 16 Jul 2010 07:47:16 GMT
1)I found one of my regionserver close with exception, why does this
happen??
2) I have 3 machine run hbase: one runs master, the other two run
regionserver, each machine run zookeeper. and I set  HBASE_HEAPSIZE=4000 and
use 64-bit jdk

3) following info are logs

2010-07-15 20:08:06,920 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87873, Hit=87762, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87367987632751%, Miss
Ratio=0.12631866848096251%, Evicted/Run=NaN

2010-07-15 20:08:11,735 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Skipping major compaction of ComFm because one (major) compacted file only
and elapsedTime 90019429ms is < ttl=9223372036854775807

2010-07-15 20:08:11,737 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Skipping major compaction of commission_attr because one (major) compacted
file only and elapsedTime 780030272ms is < ttl=9223372036854775807

2010-07-15 20:08:11,737 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Skipping major compaction of price because one (major) compacted file only
and elapsedTime 100019204ms is < ttl=9223372036854775807

2010-07-15 20:08:11,739 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Skipping major compaction of info because one (major) compacted file only
and elapsedTime 800029761ms is < ttl=9223372036854775807

2010-07-15 20:08:11,739 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Skipping major compaction of commission_attr because one (major) compacted
file only and elapsedTime 780030215ms is < ttl=9223372036854775807

2010-07-15 20:09:06,921 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87875, Hit=87764, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87368583679199%, Miss
Ratio=0.1263157930225134%, Evicted/Run=NaN

2010-07-15 20:10:06,919 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87877, Hit=87766, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87368583679199%, Miss
Ratio=0.12631291756406426%, Evicted/Run=NaN

2010-07-15 20:11:06,921 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87879, Hit=87768, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87369179725647%, Miss
Ratio=0.12631004210561514%, Evicted/Run=NaN

2010-07-15 20:12:06,919 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87881, Hit=87770, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87369179725647%, Miss
Ratio=0.126307166647166%, Evicted/Run=NaN

2010-07-15 20:13:06,924 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87883, Hit=87772, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87369775772095%, Miss
Ratio=0.1263042911887169%, Evicted/Run=NaN

2010-07-15 20:14:06,918 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87903, Hit=87792, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87372159957886%, Miss
Ratio=0.12627554824575782%, Evicted/Run=NaN

2010-07-15 20:15:06,924 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87925, Hit=87814, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87375736236572%, Miss
Ratio=0.126243953127414%, Evicted/Run=NaN

2010-07-15 20:16:06,918 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87927, Hit=87816, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.87375736236572%, Miss
Ratio=0.12624108931049705%, Evicted/Run=NaN

2010-07-15 20:17:06,923 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=8.987755MB (9424344), Free=787.68726MB (825949928), Max=796.675MB
(835374272), Counts: Blocks=93, Access=87929, Hit=87818, Miss=111,
Evictions=0, Evicted=0, Ratios: Hit Ratio=99.8737633228302%, Miss
Ratio=0.12623821385204792%, Evicted/Run=NaN

2010-07-15 20:17:31,260 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 295729663 milliseconds - retrying

2010-07-15 20:17:33,725 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x29815d71ca0000 to sun.nio.ch.SelectionKeyImpl@679bfb30

java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]

	at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)

	at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)

2010-07-15 20:17:33,833 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null

2010-07-15 20:17:34,284 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1

org.apache.hadoop.hbase.Leases$LeaseStillHeldException

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)

	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)

	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

	at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteE
xceptionHandler.java:94)

	at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptio
nHandler.java:48)

	at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExcept
ionHandler.java:66)

	at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:54
9)

	at java.lang.Thread.run(Thread.java:619)

2010-07-15 20:17:34,781 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server linux72/10.137.97.72:2222

2010-07-15 20:17:34,782 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected
local=/10.137.97.73:23381 remote=linux72/10.137.97.72:2222]

2010-07-15 20:17:34,782 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful

2010-07-15 20:17:34,783 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x29815d71ca0000 to sun.nio.ch.SelectionKeyImpl@10606b15

java.io.IOException: Session Expired

	at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java
:589)

	at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)

	at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)

2010-07-15 20:17:34,783 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null

2010-07-15 20:17:34,783 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired

2010-07-15 20:17:34,784 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=11, stores=11, storefiles=8, storefileIndexSize=0,
memstoreSize=0, compactionQueueSize=0, usedHeap=35, maxHeap=3983,
blockCacheSize=9424344, blockCacheFree=825949928, blockCacheCount=93,
blockCacheHitRatio=99, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0

2010-07-15 20:17:34,996 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
295738661ms, ten times longer than scheduled: 10000

2010-07-15 20:17:34,997 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/10.137.97.73:60020.logFlusher exiting

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting

2010-07-15 20:17:35,287 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 15 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 60020: exiting

2010-07-15 20:17:35,288 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting

2010-07-15 20:17:35,289 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 60020: exiting

2010-07-15 20:17:35,289 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 60020: exiting

2010-07-15 20:17:35,289 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 60020: exiting

2010-07-15 20:17:35,289 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60020: exiting

2010-07-15 20:17:35,289 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 11 on 60020: exiting

2010-07-15 20:17:35,288 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting

2010-07-15 20:17:35,395 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.

2010-07-15 20:17:35,396 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/10.137.97.73:60020.compactor exiting

2010-07-15 20:17:35,396 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/10.137.97.73:60020.majorCompactionChecker exiting

2010-07-15 20:17:35,396 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing hlog writer in
hdfs://linux71:9000/hbase/.logs/linux73,60020,1277775660008

2010-07-15 20:17:35,396 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/10.137.97.73:60020.cacheFlusher exiting

2010-07-15 20:17:35,402 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/linux73,60020,1277775660008/hlog.dat.1279194737171 File does
not exist. Holder DFSClient_-189304969 does not have any open files.

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1332)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1323)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNam
esystem.java:1251)

	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)

	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

	at java.lang.reflect.Method.invoke(Method.java:597)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:396)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)



	at org.apache.hadoop.ipc.Client.call(Client.java:740)

	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)

	at $Proxy1.addBlock(Unknown Source)

	at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)

	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

	at java.lang.reflect.Method.invoke(Method.java:597)

	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocati
onHandler.java:82)

	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHand
ler.java:59)

	at $Proxy1.addBlock(Unknown Source)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSCli
ent.java:2937)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSCl
ient.java:2819)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:
2102)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
java:2288)



2010-07-15 20:17:35,402 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null

2010-07-15 20:17:35,402 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
block locations. Source file
"/hbase/.logs/linux73,60020,1277775660008/hlog.dat.1279194737171" -
Aborting...

2010-07-15 20:17:35,408 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort

org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/linux73,60020,1277775660008/hlog.dat.1279194737171 File does
not exist. Holder DFSClient_-189304969 does not have any open files.

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1332)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1323)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNam
esystem.java:1251)

	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)

	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

	at java.lang.reflect.Method.invoke(Method.java:597)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:396)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)



	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

	at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)

	at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)

	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

	at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteE
xceptionHandler.java:94)

	at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptio
nHandler.java:48)

	at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:61
3)

	at java.lang.Thread.run(Thread.java:619)

2010-07-15 20:17:35,411 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
ASIncreSampleRelation,,1277362488557

2010-07-15 20:17:35,411 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing ASIncreSampleRelation,,1277362488557: disabling compactions &
flushes

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
ASIncreSampleRelation,,1277362488557

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region ASIncreSampleRelation,,1277362488557

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,412 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed ASIncreSampleRelation,,1277362488557

2010-07-15 20:17:35,412 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
DataVersion,,1277362490763

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing DataVersion,,1277362490763: disabling compactions & flushes

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
DataVersion,,1277362490763

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region DataVersion,,1277362490763

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,412 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed DataVersion,,1277362490763

2010-07-15 20:17:35,412 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
ASSampleRelation,,1277362486181

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing ASSampleRelation,,1277362486181: disabling compactions & flushes

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
ASSampleRelation,,1277362486181

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region ASSampleRelation,,1277362486181

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,412 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed ASSampleRelation,,1277362486181

2010-07-15 20:17:35,412 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
IncreContract,,1277362521297

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing IncreContract,,1277362521297: disabling compactions & flushes

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
IncreContract,,1277362521297

2010-07-15 20:17:35,412 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region IncreContract,,1277362521297

2010-07-15 20:17:35,413 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,413 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed IncreContract,,1277362521297

2010-07-15 20:17:35,413 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
AllocatedContract2,,1277362481780

2010-07-15 20:17:35,413 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing AllocatedContract2,,1277362481780: disabling compactions & flushes

2010-07-15 20:17:35,413 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
AllocatedContract2,,1277362481780

2010-07-15 20:17:35,413 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region AllocatedContract2,,1277362481780

2010-07-15 20:17:35,413 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,414 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed AllocatedContract2,,1277362481780

2010-07-15 20:17:35,414 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
network_attr,,1277362514267

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing network_attr,,1277362514267: disabling compactions & flushes

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
network_attr,,1277362514267

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region network_attr,,1277362514267

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed commission_attr

2010-07-15 20:17:35,414 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed network_attr,,1277362514267

2010-07-15 20:17:35,414 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
category_rate_card,,1277362499716

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing category_rate_card,,1277362499716: disabling compactions & flushes

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
category_rate_card,,1277362499716

2010-07-15 20:17:35,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region category_rate_card,,1277362499716

2010-07-15 20:17:35,417 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed price

2010-07-15 20:17:35,417 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed category_rate_card,,1277362499716

2010-07-15 20:17:35,417 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
IncreAllocatedContractDel,,1277362495298

2010-07-15 20:17:35,417 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing IncreAllocatedContractDel,,1277362495298: disabling compactions &
flushes

2010-07-15 20:17:35,417 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
IncreAllocatedContractDel,,1277362495298

2010-07-15 20:17:35,417 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region
IncreAllocatedContractDel,,1277362495298

2010-07-15 20:17:35,417 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,418 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed IncreAllocatedContractDel,,1277362495298

2010-07-15 20:17:35,418 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
AllocatedContract1,,1277362481376

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing AllocatedContract1,,1277362481376: disabling compactions & flushes

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
AllocatedContract1,,1277362481376

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region AllocatedContract1,,1277362481376

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed ComFm

2010-07-15 20:17:35,418 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed AllocatedContract1,,1277362481376

2010-07-15 20:17:35,418 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region -ROOT-,,0

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing -ROOT-,,0: disabling compactions & flushes

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on -ROOT-,,0

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region -ROOT-,,0

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed info

2010-07-15 20:17:35,418 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed -ROOT-,,0

2010-07-15 20:17:35,418 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
platform_attr,,1277362509964

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing platform_attr,,1277362509964: disabling compactions & flushes

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
platform_attr,,1277362509964

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region platform_attr,,1277362509964

2010-07-15 20:17:35,418 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed commission_attr

2010-07-15 20:17:35,418 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed platform_attr,,1277362509964

2010-07-15 20:17:35,419 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
10.137.97.73:60020

2010-07-15 20:17:41,974 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting

2010-07-15 20:17:41,975 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x29815d71ca0000

2010-07-15 20:17:41,975 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x29815d71ca0000

2010-07-15 20:17:41,975 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x29815d71ca0000

2010-07-15 20:17:41,975 INFO org.apache.zookeeper.ZooKeeper: Session:
0x29815d71ca0000 closed

2010-07-15 20:17:41,975 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection with
ZooKeeper

2010-07-15 20:17:41,975 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down

2010-07-15 20:17:41,976 ERROR org.apache.hadoop.hdfs.DFSClient: Exception
closing file /hbase/.logs/linux73,60020,1277775660008/hlog.dat.1279194737171
: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/linux73,60020,1277775660008/hlog.dat.1279194737171 File does
not exist. Holder DFSClient_-189304969 does not have any open files.

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1332)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1323)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNam
esystem.java:1251)

	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)

	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

	at java.lang.reflect.Method.invoke(Method.java:597)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:396)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)



org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/linux73,60020,1277775660008/hlog.dat.1279194737171 File does
not exist. Holder DFSClient_-189304969 does not have any open files.

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1332)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1323)

	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNam
esystem.java:1251)

	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)

	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

	at java.lang.reflect.Method.invoke(Method.java:597)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:396)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)



	at org.apache.hadoop.ipc.Client.call(Client.java:740)

	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)

	at $Proxy1.addBlock(Unknown Source)

	at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)

	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

	at java.lang.reflect.Method.invoke(Method.java:597)

	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocati
onHandler.java:82)

	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHand
ler.java:59)

	at $Proxy1.addBlock(Unknown Source)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSCli
ent.java:2937)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSCl
ient.java:2819)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:
2102)

	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
java:2288)

2010-07-15 20:17:42,080 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
regionserver/10.137.97.73:60020 exiting

2010-07-15 20:17:44,323 INFO org.apache.hadoop.hbase.Leases:
regionserver/10.137.97.73:60020.leaseChecker closing leases

2010-07-15 20:17:44,324 INFO org.apache.hadoop.hbase.Leases:
regionserver/10.137.97.73:60020.leaseChecker closed leases

2010-07-15 20:17:44,325 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.

2010-07-15 20:17:44,325 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete

Mime
View raw message