hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eason.Lee" <leongf...@gmail.com>
Subject Hbase can't stop after I killed a job importing data into hbase.
Date Wed, 18 Nov 2009 08:07:12 GMT
I am importing data into hbase table
and I killed the job for I found there is something wrong with records
So i want to disable the table and import data again,but i can't disable it
Error:
disable 'xhs2'
NativeException: org.apache.hadoop.hbase.RegionException: Retries exhausted,
it took too long to wait for the table xhs2 to be disabled.
        from org/apache/hadoop/hbase/client/HBaseAdmin.java:411:in
`disableTable'
        from org/apache/hadoop/hbase/client/HBaseAdmin.java:370:in
`disableTable'
        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
        from java/lang/reflect/Method.java:597:in `invoke'
        from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
        from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call'
        from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
        from org/jruby/ast/CallOneArgNode.java:57:in `interpret'
        from org/jruby/ast/NewlineNode.java:104:in `interpret'
        from org/jruby/ast/BlockNode.java:71:in `interpret'
        from
org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
        from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in
`call'
        from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
... 106 levels...
        from
home/hadoop/workbase/hbase_minus_0_dot_20_dot_1/bin/$_dot_dot_/bin/hirb#start:-1:in
`call'
        from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
`call'
        from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in
`call'
        from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
`call'
        from org/jruby/runtime/callsite/CachingCallSite.java:253:in
`cacheAndCall'
        from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
        from
home/hadoop/workbase/hbase_minus_0_dot_20_dot_1/bin/$_dot_dot_/bin/hirb.rb:497:in
`__file__'
        from
home/hadoop/workbase/hbase_minus_0_dot_20_dot_1/bin/$_dot_dot_/bin/hirb.rb:-1:in
`load'
        from org/jruby/Ruby.java:577:in `runScript'
        from org/jruby/Ruby.java:480:in `runNormally'
        from org/jruby/Ruby.java:354:in `runFromMain'
        from org/jruby/Main.java:229:in `run'
        from org/jruby/Main.java:110:in `run'
        from org/jruby/Main.java:94:in `main'
        from /home/hadoop/workbase/hbase-0.20.1/bin/../bin/hirb.rb:350:in
`disable'
        from (hbase):3hbase(main):003:0> quit

So i try to restart the hbase and try again
but i found that i can't stop the hbase~~
Wed Nov 18 14:51:09 EST 2009 Stopping master
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=hadoop1.trs
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_14
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/home/hadoop/media/jdk1.6.0_14/jre
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/hadoop/workbase/hbase-0.20.1/bin/../conf:/home/hadoop/media/jdk1.6.0_14/lib/tools.jar:/home/hadoop/workbase/hbase-0.20.1/bin/..:/home/hadoop/workbase/hbase-0.20.1/bin/../hbase-0.20.1.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../hbase-0.20.1-test.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/AgileJSON-2009-03-30.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/commons-el-from-jetty-5.1.4.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/commons-httpclient-3.0.1.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/commons-logging-api-1.0.4.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/commons-math-1.1.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/hadoop-0.20.1-hdfs127-core.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/hadoop-0.20.1-test.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jasper-compiler-5.5.12.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jasper-runtime-5.5.12.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jetty-6.1.14.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jetty-util-6.1.14.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jruby-complete-1.2.0.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/json.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/junit-3.8.1.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/libthrift-r771587.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/log4j-1.2.15.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/lucene-core-2.2.0.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/xmlenc-0.52.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/zookeeper-3.2.1.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jsp-2.1/jsp-2.1.jar:/home/hadoop/workbase/hbase-0.20.1/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/home/hadoop/workbae/hadoop-0.20.1/conf:/home/hadoop/workbase/hadoop-0.20.1/hadoop-0.20.1-core.jar
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/home/hadoop/workbase/hbase-0.20.1/bin/../lib/native/Linux-amd64-64
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2009-11-18 14:51:10,296 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2009-11-18 14:51:10,297 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.16.60-0.21-smp
2009-11-18 14:51:10,297 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hadoop
2009-11-18 14:51:10,297 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/hadoop
2009-11-18 14:51:10,297 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/home/hadoop/workbase/hbase-0.20.1
2009-11-18 14:51:10,297 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
connectString=hadoop4.site:2181,hadoop3.site:2181,hadoop2.site:2181,hadoop1.trs:2181,hadoop5.site:2181
sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@19d009b4
2009-11-18 14:51:10,298 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2009-11-18 14:51:10,314 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server hadoop5.site/192.9.200.238:2181
2009-11-18 14:51:10,315 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.9.200.164:51576 remote=hadoop5.site/192.9.200.238:2181]
2009-11-18 14:51:10,316 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-11-18 14:51:10,429 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
shutdown requested. Starting to quiesce servers
2009-11-18 14:51:10,432 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set state node
in ZooKeeper
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /hbase/shutdown
        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:699)
        at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.setClusterState(ZooKeeperWrapper.java:283)
        at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:735)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
hadoop@hadoop1:~/workbase/hbase-0.20.1/logs> tail
hbase-hadoop-master-hadoop1.log -n 100
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-11-18 14:36:50,964 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE:
xhs2,hdfs:\x2F\x2Fhadoop1.trs:9000\x2Fuser\x2Fhadoop\x2Fxhsxml\x2Fxhs_1.xml_204395461,1258569632372
from hadoop4.site,60020,1258569462903; 1 of 1
2009-11-18 14:36:50,964 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000, call regionServerReport(address: 192.9.200.239:60020,
startcode: 1258569462903, load: (requests=0, regions=1, usedHeap=393,
maxHeap=1983), [Lorg.apache.hadoop.hbase.HMsg;@1ec5b819,
[Lorg.apache.hadoop.hbase.HRegionInfo;@46b98117) from 192.9.200.239:38357:
error: java.io.IOException: java.lang.IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close or pending
open. State:
name=xhs2,hdfs://hadoop1.trs:9000/user/hadoop/xhsxml/xhs_1.xml_204395461,1258569632372,
unassigned=false, pendingOpen=false, open=false, closing=true,
pendingClose=false, closed=false, offlined=true
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to
be closed if it was not already marked as pending close or pending open.
State:
name=xhs2,hdfs://hadoop1.trs:9000/user/hadoop/xhsxml/xhs_1.xml_204395461,1258569632372,
unassigned=false, pendingOpen=false, open=false, closing=true,
pendingClose=false, closed=false, offlined=true
        at
org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1603)
        at
org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1046)
        at
org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:658)
        at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:453)
        at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
        at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
        at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:720)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-11-18 14:36:53,971 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE:
xhs2,hdfs:\x2F\x2Fhadoop1.trs:9000\x2Fuser\x2Fhadoop\x2Fxhsxml\x2Fxhs_1.xml_204395461,1258569632372
from hadoop4.site,60020,1258569462903; 1 of 1
2009-11-18 14:36:53,971 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000, call regionServerReport(address: 192.9.200.239:60020,
startcode: 1258569462903, load: (requests=0, regions=1, usedHeap=393,
maxHeap=1983), [Lorg.apache.hadoop.hbase.HMsg;@7e694b7e,
[Lorg.apache.hadoop.hbase.HRegionInfo;@32db7e08) from 192.9.200.239:38357:
error: java.io.IOException: java.lang.IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close or pending
open. State:
name=xhs2,hdfs://hadoop1.trs:9000/user/hadoop/xhsxml/xhs_1.xml_204395461,1258569632372,
unassigned=false, pendingOpen=false, open=false, closing=true,
pendingClose=false, closed=false, offlined=true
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to
be closed if it was not already marked as pending close or pending open.
State:
name=xhs2,hdfs://hadoop1.trs:9000/user/hadoop/xhsxml/xhs_1.xml_204395461,1258569632372,
unassigned=false, pendingOpen=false, open=false, closing=true,
pendingClose=false, closed=false, offlined=true
        at
org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1603)
        at
org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1046)
        at
org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:658)
        at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:453)
        at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
        at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
        at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:720)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

any suggestion would be help~~
thanks~~

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message