Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.160.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAFAd71VTd4_-p0jG7Gmqmn-JMMM4uYg=z0QJgu0ki9wAjHvRow@mail.gmail.com>
References: 
 <CAFAd71VTd4_-p0jG7Gmqmn-JMMM4uYg=z0QJgu0ki9wAjHvRow@mail.gmail.com>
Date: Fri, 31 Oct 2014 07:56:03 -0700
Message-ID: 
 <CALte62zKi3Bj0SOhS+xSgU8hEpe5GpC0xjjr-Xz7AC+mcERTbA@mail.gmail.com>
Subject: Re: can't start hbase.
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11399b481fe7710506b93257

--001a11399b481fe7710506b93257
Content-Type: text/plain; charset=UTF-8

bq.  Found 1 replicas but expecting no less than 3 replicas.

This is controlled by "hbase.regionserver.hlog.tolerable.lowreplication"
You can lower the value of the above config.

Cheers

On Thu, Oct 30, 2014 at 10:16 PM, Li Li <fancyerii@gmail.com> wrote:

> hi all,
>    I am using hbase and also phoenix(some tables are managed by myself
> and some are created by phoenix).
>   Last night, the disk is full . I killed the hbase
> and hadoop related processes. But After that I can't start hbase
> anymore.
> I am using ubuntu 12.04 and hadoop-1.2.1 and hbase 0.98.5 with phoenix
> 4.1.0
> The region server print error like:
>
> 2014-10-31 12:33:49,835 INFO
> [mobvoi-knowledge-graph-0,60020,1414729587199-recovery-writer--pool4-t2]
> client.AsyncProcess: #9, waiting for some tasks to finish. Expected
> max=0, tasksSent=31, tasksDone=30, currentTasksDone=30, retries=30
> hasError=false, tableName=BAIDUMUSIC.BAIDUMUSIC_IDX
>
> 2014-10-31 12:33:49,842 INFO  [htable-pool9-t1] client.AsyncProcess:
> #9, table=BAIDUMUSIC.BAIDUMUSIC_IDX, attempt=31/350 failed 1 ops, last
> exception: org.apache.hadoop.hbase.exceptions.RegionOpeningException:
> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
>
> BAIDUMUSIC.BAIDUMUSIC_IDX,\x0D\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1414488087936.69e9a7efb9ee00b1ecfe50f825e7cc5b.
> is opening on mobvoi-knowledge-graph-0,60020,1414729587199
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2692)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4139)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3363)
>
>         at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
>
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
>
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>  on mobvoi-knowledge-graph-0,60020,1414484323072, tracking started Fri
> Oct 31 12:26:39 CST 2014, retrying after 20119 ms, replay 1 ops.
>
> 2014-10-31 12:33:55,747 INFO
> [mobvoi-knowledge-graph-0,60020,1414729587199-recovery-writer--pool4-t3]
> client.AsyncProcess: #25, waiting for some tasks to finish. Expected
> max=0, tasksSent=31, tasksDone=30, currentTasksDone=30, retries=30
> hasError=false, tableName=BAIDUMUSIC.BAIDUMUSIC_IDX
>
> 2014-10-31 12:33:55,755 INFO  [htable-pool24-t1] client.AsyncProcess:
> #25, table=BAIDUMUSIC.BAIDUMUSIC_IDX, attempt=31/350 failed 2 ops,
> last exception:
> org.apache.hadoop.hbase.exceptions.RegionOpeningException:
> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
>
> BAIDUMUSIC.BAIDUMUSIC_IDX,\x0C\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1414488087936.378bef2d741a9c00761160670329fc7c.
> is opening on mobvoi-knowledge-graph-0,60020,1414729587199
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2692)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4139)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3363)
>
>         at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
>
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
>
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>  on mobvoi-knowledge-graph-0,60020,1414484323072, tracking started Fri
> Oct 31 12:26:45 CST 2014, retrying after 20001 ms, replay 2 ops.
>
> I also find warns like
> wal.FSHLog: HDFS pipeline error detected. Found 1 replicas but
> expecting no less than 3 replicas.  Requesting close of hlog.
> But my hbase cluster is pesudo cluster that all processes are running
> in a single machine and the hdfs (I have set dfs.replication=1 in
> hdfs-site.xml) I have tried to set replication factor to 1 by hadoop
> fs -setrep -R 1 / but no luck
>

--001a11399b481fe7710506b93257--