Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: Pankaj kr <pankaj.kr@huawei.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: RE: Table regions are written in wrong path in HDFS
Thread-Topic: Table regions are written in wrong path in HDFS
Thread-Index: AdEkSz1YqIgSt3yASaSAc82jDseL2P//s+WA//0klrA=
Date: Mon, 23 Nov 2015 02:02:46 +0000
Message-ID: 
 <74ECFFA8DC3B6847888649793C770FE0A21B61C6@szxeml559-mbx.china.huawei.com>
References: 
 <74ECFFA8DC3B6847888649793C770FE0A21AFDF0@szxeml559-mbs.china.huawei.com>
 <8001A95F-74F0-4FFC-91F9-C1B171D386A6@gmail.com>
In-Reply-To: <8001A95F-74F0-4FFC-91F9-C1B171D386A6@gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks Ted,=20

All regions are in the same condition.=20
Even the table descriptor also,

drwxr-xr-x   - hbase hadoop          0 2015-11-18 00:04 /hbase/data/default=
/TestTable/TestTable/.tabledesc
-rw-r--r--   3 hbase hadoop        289 2015-11-18 00:04 /hbase/data/default=
/TestTable/TestTable/.tabledesc/.tableinfo.0000000001

So restarting HM/RS doesn't help.

Regards,
Pankaj

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]=20
Sent: 21 November 2015 22:24
To: user@hbase.apache.org
Subject: Re: Table regions are written in wrong path in HDFS

Can you trace this region through master / region server log to see if ther=
e is some clue ?

Cheers

> On Nov 21, 2015, at 2:56 AM, Pankaj kr <pankaj.kr@huawei.com> wrote:
>=20
> Hi Folks,
>=20
> We met a very weird scenario.
> We are running PE tool, during testing we found all regions are in transi=
tion in state FAILED_OPEN.
>=20
> Region server are failed to open the regions with below exception,
> 2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed=
 open of region=3DTestTable,00000000000000000025000000,1447776261671.d5b64a=
d96deb3b47467db97669c009fb., starting to roll back the global memstore size=
. | org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegi=
on(OpenRegionHandler.java:386)
> java.lang.IllegalStateException: Could not instantiate a region instance.
>                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion=
(HRegion.java:6229)
>                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegio=
n(HRegion.java:6536)
>                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegio=
n(HRegion.java:6508)
>                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegio=
n(HRegion.java:6464)
>                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegio=
n(HRegion.java:6415)
>                at org.apache.hadoop.hbase.regionserver.handler.OpenRegion=
Handler.openRegion(OpenRegionHandler.java:363)
>                at org.apache.hadoop.hbase.regionserver.handler.OpenRegion=
Handler.process(OpenRegionHandler.java:129)
>                at org.apache.hadoop.hbase.executor.EventHandler.run(Event=
Handler.java:128)
>                at java.util.concurrent.ThreadPoolExecutor.runWorker(Threa=
dPoolExecutor.java:1142)
>                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre=
adPoolExecutor.java:617)
>                at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
>                at sun.reflect.GeneratedConstructorAccessor17.newInstance(=
Unknown Source)
>                at sun.reflect.DelegatingConstructorAccessorImpl.newInstan=
ce(DelegatingConstructorAccessorImpl.java:45)
>                at java.lang.reflect.Constructor.newInstance(Constructor.j=
ava:422)
>                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion=
(HRegion.java:6226)
>                ... 10 more
> Caused by: java.lang.IllegalArgumentException: Need table descriptor
>                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRe=
gion.java:695)
>                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRe=
gion.java:672)
>                ... 14 more
>=20
> In HDFS, Table region was written as below,
> drwxr-xr-x   - hbase hadoop          0 2015-11-18 00:05 /hbase/data/defau=
lt/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
> -rw-r--r--   3 hbase hadoop         96 2015-11-18 00:05 /hbase/data/defau=
lt/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
>=20
> Here table name "TestTable" has come twice, how is that possible?
>=20
> Have anyone met this scenario?
> We are using HBase 1.0.2 and HDFS 2.7.2 version.
>=20
> Any suggestion/help would be much appreciated.
>=20
> Regards,
> Pankaj