hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Unable to create table
Date Tue, 29 Mar 2011 18:51:48 GMT
There's a reason why disabling takes time, if you delete rows from
.META. you might end up in an inconsistent situation and we'll have a
hard time helping you :)

So HBASE-3557 is what you want.

Regarding your current issue, RIT is "region in transition" meaning
that the region is in a state recognized by the master as either
moving from one region server to another, or just closing. Normally
after disabling a table its regions all left their in transition
state, but by deleting the rows directly in .META. who knows exactly
what's the state of your table?

J-D

On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
<hsreekumar@clickable.com> wrote:
> Hi J-D,
>
> Here is the tail of HMaster log:
>
> 2011-03-29 23:48:51,155 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:52,158 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:53,161 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:54,163 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:55,164 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:56,166 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:57,168 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:58,170 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:48:59,172 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:49:00,174 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:49:01,176 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:49:02,178 DEBUG
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
>  region to clear regions in transition;
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
> ts=1301422405271
> 2011-03-29 23:49:02,178 ERROR
> org.apache.hadoop.hbase.master.handler.TableEventHandler: Error manipulating
> table AcContact
> java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
> region to leave region
> AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions
>        at
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
>        at
> org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
>        at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>
> What is "clearing regions in transition"? What could be the issue?
>
> I was told before too that deleting from META isn't recommended. But we were
> facing the disable problem way too often. So we thought we'd make it
> in-built. What other alternatives do I have?
>
> Thx,
> Hari
>
> On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> The 60 secs timeout means that the client was waiting on the master
>> for some operation but the master took longer than 60 secs to do it,
>> so its log should be the next place too look for something whack.
>>
>> BTW deleting the rows from .META. directly is probably the worst thing
>> you can do.
>>
>> J-D
>>
>> On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
>> <hsreekumar@clickable.com> wrote:
>> > Here is the stack trace:
>> >
>> > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection,
>> > connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
>> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to
>> > server hadoop2/192.168.1.111:2181
>> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
>> established
>> > to hadoop2/192.168.1.111:2181, initiating session
>> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
>> complete
>> > on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
>> > negotiated timeout = 180000
>> > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
>> > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
>> > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries
>> from
>> > META. Retrying to create table
>> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table
>> even
>> > after Cleaning Meta entries
>> > 11/03/28 18:48:02 FATAL create.CreateTableByXML:
>> > *************************************************************
>> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
>> > 192.168.1.57:60000 failed on socket timeout exception:
>> > java.net.SocketTimeoutException: 60000 millis timeout while waiting for
>> > channel to be ready for read. ch :
>> java.nio.channels.SocketChannel[connected
>> > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
>> > 11/03/28 18:48:02 FATAL create.CreateTableByXML:
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> > $Proxy4.createTable(Unknown Source)
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
>> >
>> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
>> >
>> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
>> > Source)
>> > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
>> Source)
>> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > java.lang.reflect.Method.invoke(Method.java:597)
>> > org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> >
>> > The "CleanFromMeta" function catches IOException and deletes all rows
>> from
>> > .META. We had added this in the exception catch block because we used to
>> > face the "Table taking too long to be disabled" exception often. It seems
>> > the rows in META already get created when the IOException is thrown.
>> > CleanFromMeta cleans .META. and then I try again to create the table,
>> after
>> > which I get the socket timeout exception.
>> >
>> > I am using Hadoop v0.20.2, r911707. Can this be a reason for this error?
>> I
>> > get the message "You are currently running the HMaster without HDFS
>> append
>> > support enabled. This may result in data loss. Please see the HBase
>> > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
>> > details." on the HBase Master UI.
>> >
>> > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
>> > <hsreekumar@clickable.com>wrote:
>> >
>> >> Hi Stack
>> >>
>> >> yes the tablename is AcContact. The tableName variable was wrong.
>> >> Fixed it now but I still get the same error. Schema is just something
>> >> created by parsing an XML file which has stuff like column family
>> >> name, compression type etc so I guess it doesn't have much to do with
>> >> version. Except that I had to change the bloom filter variable to
>> >> String ( used to be boolean in 0.20.6). I will paste the stack trace
>> >> asap
>> >>
>> >> hari
>> >>
>> >> On Monday, March 28, 2011, Stack <stack@duboce.net> wrote:
>> >> > Can I see more of the stack track please Hari and is AcContact the
>> >> > table you are creating?  Is the schema you've saved aside one you
>> >> > created with 0.20 hbase?  I don't think it matters but asking just
in
>> >> > case.
>> >> > St.Ack
>> >> >
>> >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
>> >> > <hsreekumar@clickable.com> wrote:
>> >> >> Hi,
>> >> >>
>> >> >> I am trying to create table in hbase v0.90.1 and I get the following
>> >> error:
>> >> >>
>> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
>> connection
>> >> to
>> >> >> server hadoop2/192.168.1.111:2181
>> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
>> >> established
>> >> >> to hadoop2/192.168.1.111:2181, initiating s
>> >> >> ession
>> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
>> >> complete
>> >> >> on server hadoop2/192.168.1.111:2181, sess
>> >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000
>> >> >> 11/03/28 18:39:52 INFO
>> >> client.HConnectionManager$HConnectionImplementation:
>> >> >> Closed zookeeper sessionid=0x12efc946d6600
>> >> >> 0b
>> >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session:
>> 0x12efc946d66000b
>> >> >> closed
>> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
>> >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException:
>> No
>> >> >> server address listed in .META. for region
>> >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
>> >> creating
>> >> >> table: Table1
>> >> >>
>> >> >> This is the code I am using:
>> >> >> .....
>> >> >> HTableDescriptor desc =
>> >> CreateTableByXML.convertSchemaToDescriptor(schema);
>> >> >>    try {
>> >> >>      hbaseAdmin.createTable(desc);
>> >> >>    } catch (IOException e) {
>> >> >>      CreateTableByXML.LOG.error("Caught IOException: " +
>> e.getMessage()
>> >> >>              + " while creating table: " + tableName);
>> >> >> ....
>> >> >> .....
>> >> >>
>> >> >>
>> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1 and
it
>> >> started
>> >> >> giving this error. Any ideas?
>> >> >>
>> >> >> Hari
>> >> >>
>> >> >
>> >>
>> >
>>
>

Mime
View raw message