hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Regions won't come online
Date Mon, 12 Oct 2015 15:26:16 GMT
Have you checked master to see if region assignment went okay ?

Cheers 

> On Oct 12, 2015, at 7:56 AM, Jurian Broertjes <jurian.broertjes@openindex.io> wrote:
> 
> Hi all,
> 
> I'm using hbase (1.1.2) with phoenix (4.5.2-HBase-1.1) and had some (minor) HDFS issues.
The HDFS issues are resolved and when I try to bring HBase back up, I run into issues where
some regions won't come online.
> 
> Some RS log:
> 2015-10-12 14:08:54,681 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t1]
client.AsyncProcess: #8, waiting for 1  actions to finish
> 2015-10-12 14:08:55,781 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t2]
client.AsyncProcess: #13, waiting for 1  actions to finish
> 2015-10-12 14:08:55,819 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t3]
client.AsyncProcess: #15, waiting for 2  actions to finish
> 2015-10-12 14:08:59,119 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t4]
client.AsyncProcess: #24, waiting for 1  actions to finish
> 2015-10-12 14:08:59,138 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t5]
client.AsyncProcess: #25, waiting for 2  actions to finish
> 2015-10-12 14:09:04,692 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t1]
client.AsyncProcess: #8, waiting for 1  actions to finish
> 2015-10-12 14:09:05,793 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t2]
client.AsyncProcess: #13, waiting for 1  actions to finish
> 2015-10-12 14:09:05,831 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t3]
client.AsyncProcess: #15, waiting for 2  actions to finish
> 2015-10-12 14:09:07,214 INFO [regionserver/cn1.xxx.xx/89.188.14.2:16020-shortCompactions-1444658915963]
client.AsyncProcess: #23, waiting for some tasks to finish. Expected max=0, tasksInProgress=9
> 2015-10-12 14:09:09,131 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t4]
client.AsyncProcess: #24, waiting for 1  actions to finish
> 2015-10-12 14:09:09,150 INFO [cn1.xxx.xx,16020,1444658908121-recovery-writer--pool6-t5]
client.AsyncProcess: #25, waiting for 2  actions to finish
> 2015-10-12 14:09:12,945 INFO  [htable-pool8-t1] client.AsyncProcess: #8, table=OUTLINKS_SSI_INDEX,
attempt=10/350 failed=1ops, last exception: org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region OUTLINKS_SSI_INDEX,,1440761791894.c9cfcf16be9852553efe45e36387a4b1.
is not online on cn1.xxx.xx,16020,1444658908121
>  at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1991)
>  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
>  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>  at java.lang.Thread.run(Thread.java:745)
> on cn1.xxx.xx,16020,1444654232108, tracking started null, retrying after=10086ms, replay=1ops
> 
> The cluster consists of 2 masters and 3 region servers and an external Zookeeper.
> 
> Anyone knows what's going on here?
> 
> Thanks in advance,
> 
> Jurian

Mime
View raw message