Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of
 ramkrishna.vasudevan@huawei.com designates 119.145.14.64 as permitted sender)
From: "Ramkrishna.S.Vasudevan" <ramkrishna.vasudevan@huawei.com>
To: <user@hbase.apache.org>
References: <95214007-B732-47BC-B480-4E90490294FC@gmail.com>
	<507d140c.8284440a.1e62.ffffe73dSMTPIN_ADDED@mx.google.com>
	<7A5B78A2-3278-40CF-9C7F-D4CFBF3621CE@gmail.com>
	<F4EFADCC0DC42249A948824C0935716790689A@IT-Exch-03.1verge.com>
	<507d2085.8652420a.5427.ffffa092SMTPIN_ADDED@mx.google.com>
	<BCAF02D1-CC89-4DC5-A576-81C4601D72B3@gmail.com>
 <CAGngS9c5Nu4uxuoSAsixq=ZDj3=fNA4HHuG5rcnChznKtr-LrA@mail.gmail.com>
In-Reply-To: 
 <CAGngS9c5Nu4uxuoSAsixq=ZDj3=fNA4HHuG5rcnChznKtr-LrA@mail.gmail.com>
Subject: RE: hbase can't drop a table
Date: Wed, 17 Oct 2012 09:25:16 +0530
Message-ID: <009e01cdac1b$38857fe0$a9907fa0$@vasudevan@huawei.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="gb2312"
Content-Transfer-Encoding: quoted-printable
Thread-Index: Ac2ruQVPoL2Oakz4QX+MRgN0V+foZQAYh1DA
Content-Language: en-us

Thanks for the write up Kevin.  Many users will be benefited by this.

Regards
Ram

> -----Original Message-----
> From: Kevin O'dell [mailto:kevin.odell@cloudera.com]
> Sent: Tuesday, October 16, 2012 9:42 PM
> To: user@hbase.apache.org
> Subject: Re: hbase can't drop a table
>=20
> If you get in that situation again:
>=20
> 1.) Verify that you don't have any remnants of the tables in HDFS
>      hadoop fs -ls /hbase/
>=20
> 2.) If you do have any remnants and you don't care about these tables
>      hadoop fs -mv /hbase/<table_name> /tmp
>=20
> 3.) ./bin/hbase hbck -fixMeta -fixAssignments
>=20
> This should clean up META to match your HDFS and allow you to move
> forward.
>  If not you may need a restart of HBase in case you had a RIT or
> something
> like that hanging around.
>=20
> On Tue, Oct 16, 2012 at 5:06 AM, =CC=C6 =D3=B1 <ivytang0812@gmail.com> =
wrote:
>=20
> > After checking the .META. table , the ivy test_deu and deu_ivytest
> entries
> > do exist .
> >
> > ROW                                                 COLUMN+CELL
> >  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> > column=3Dinfo:regioninfo, timestamp=3D1348821682041, value=3D{NAME =
=3D>
> > 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.',
> STARTKEY =3D>
> > '', ENDKEY
> >  4c3e0277c.                                          =3D> '', =
ENCODED
> =3D>
> > 77eb091b4753dd3b713f29c4c3e0277c,}
> >  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=3Dinfo:server,
> > timestamp=3D1350032968219, value=3DELEX-LA-WEB10:61020
> >  4c3e0277c.
> >  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> > column=3Dinfo:serverstartcode, timestamp=3D1350032968219,
> value=3D1350032549030
> >  4c3e0277c.
> >
> > ROW                                                 COLUMN+CELL
> >  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> > column=3Dinfo:regioninfo, timestamp=3D1348826121970, value=3D{NAME =
=3D>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY =3D>
> > '', ENDKEY
> >  f523fcd45.                                          =3D> '', =
ENCODED
> =3D>
> > 985d6ca9986d7d8cfaf82daf523fcd45,}
> >  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=3Dinfo:server,
> > timestamp=3D1348826122164, value=3DELEX-LA-WEB10:61020
> >  f523fcd45.
> >  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> > column=3Dinfo:serverstartcode, timestamp=3D1348826122164,
> value=3D1348648468768
> >  f523fcd45.
> >
> > And i deleted them from .META. table.Things seem Ok.
> >
> > The region server won't try to load these regions.
> >
> > Yes,it seems that the HTableDescriptor  file got deleted but the =
META
> is
> > having the entry.
> >
> >
> > Thanks!
> >
> >
> > =D4=DA =
2012-10-16=A3=AC=CF=C2=CE=E74:52=A3=AC"Ramkrishna.S.Vasudevan" <
> > ramkrishna.vasudevan@huawei.com> =D0=B4=B5=C0=A3=BA
> >
> > > What does the 'list' command show?  Does it say the table exists =
or
> not?
> > >
> > > What I can infer here is that the HTableDescriptor  file got
> deleted but
> > the
> > > META is having the entry.  Any chance of the HTD getting =
accidently
> > deleted
> > > in your cluster?
> > >
> > > The hbck tool with -fixOrphanTables should atleast try to create
> the
> > > HTableDescriptor file I suppose.  Then restart the cluster and =
then
> see
> > what
> > > happens.
> > > I will not be able to access the logs even if you add it to
> pastebin.
> >  But
> > > pls do it so that some one else who has access can look into it.
> > >
> > > Regards
> > > Ram
> > >> -----Original Message-----
> > >> From: =D5=C5=C0=DA [mailto:zhanglei@youku.com]
> > >> Sent: Tuesday, October 16, 2012 1:44 PM
> > >> To: 'user@hbase.apache.org'
> > >> Subject: RE: hbase can't drop a table
> > >>
> > >> Hope this can help you!
> > >> https://issues.apache.org/jira/browse/HBASE-
> > >> =
3432?page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment-
> > >> tabpanel&focusedCommentId=3D13418790#comment-13418790
> > >>
> > >> Fowler Zhang
> > >>
> > >> -----Original Message-----
> > >> From: =CC=C6 =D3=B1 [mailto:ivytang0812@gmail.com]
> > >> Sent: 2012=C4=EA10=D4=C216=C8=D5 16:08
> > >> To: user@hbase.apache.org
> > >> Subject: Re: hbase can't drop a table
> > >>
> > >> version 0.94.0, r8547
> > >>
> > >> And the table is ivytest_deu.
> > >>
> > >>
> > >> =D4=DA =
2012-10-16=A3=AC=CF=C2=CE=E73:58=A3=AC"Ramkrishna.S.Vasudevan"
> > >> <ramkrishna.vasudevan@huawei.com> =D0=B4=B5=C0=A3=BA
> > >>
> > >>> Which version of HBase?
> > >>>
> > >>>
> > >>> The below logs that you have attached says about a different
> table
> > >> right '
> > >>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> > >>> And the one you are trying to drop is ' ivytest_deu=A1=AF
> > >>>
> > >>> Regards
> > >>> Ram
> > >>>
> > >>>
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: =CC=C6 =D3=B1 [mailto:ivytang0812@gmail.com]
> > >>>> Sent: Tuesday, October 16, 2012 1:23 PM
> > >>>> To: user@hbase.apache.org
> > >>>> Subject: hbase can't drop a table
> > >>>>
> > >>>> I disable this table ivytest_deu , drop it .Error occurs.
> > >>>>
> > >>>>
> > >>>> ERROR: java.io.IOException: java.io.IOException:
> HTableDescriptor
> > >>>> missing for ivytest_deu
> > >>>>    at
> > >>>>
> > >>
> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
> > >>>> ri
> > >>>> ptor(TableEventHandler.java:174)
> > >>>>    at
> > >>>>
> > >>
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
> > >>>> te
> > >>>> TableHandler.java:44)
> > >>>>    at
> > >>>>
> > >>
> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> > >>>>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > >>>>    at
> > >>>>
> > >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> > >>>> ja
> > >>>> va:39)
> > >>>>    at
> > >>>>
> > >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> > >>>> so
> > >>>> rImpl.java:25)
> > >>>>    at java.lang.reflect.Method.invoke(Method.java:597)
> > >>>>    at
> > >>>>
> > >>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
> > >>>> En
> > >>>> gine.java:364)
> > >>>>    at
> > >>>>
> > >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> > >>>> 13
> > >>>> 76)
> > >>>>
> > >>>> Here is some help for this command:
> > >>>> Drop the named table. Table must first be disabled. If table =
has
> > >> more
> > >>>> than one region, run a major compaction on .META.:
> > >>>>
> > >>>> hbase> major_compact ".META."
> > >>>>
> > >>>> The major_compact ".META." doesn't work.
> > >>>> Then i try to create it ,but HBase says it .
> > >>>>
> > >>>> ERROR: Table already exists: ivytest_deu!
> > >>>>
> > >>>> After checking the region server log , the region server is
> always
> > >>>> trying to load this region.
> > >>>>
> > >>>> 2012-10-16 00:00:00,308 INFO
> > >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received
> request
> > >>>> to open region:
> > >>>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > >>>> 2012-10-16 00:00:00,313 WARN
> > >>>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following
> > >> folder
> > >>>> is in HBase's root directory and doesn't contain a table
> descriptor,
> > >>>> do consider deleting it: deu_ivytest
> > >>>> 2012-10-16 00:00:00,358 DEBUG
> > >>>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
> {NAME
> > >>>> =3D>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > >>>> STARTKEY =3D> '', ENDKEY =3D> '', ENCODED =3D>
> > >>>> 985d6ca9986d7d8cfaf82daf523fcd45,}
> > >>>> 2012-10-16 00:00:00,358 DEBUG
> > >>>> org.apache.hadoop.hbase.regionserver.HRegion: Registered
> protocol
> > >>>> handler:
> > >>>>
> region=3Ddeu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > >>>>
> > >>
> =
protocol=3Dcom.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
> > >>>> eg
> > >>>> ationProtocol
> > >>>> 2012-10-16 00:00:00,358 ERROR
> > >>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> > >>>> Failed open of
> > >>>>
> region=3Ddeu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> > >>>> starting to roll back the global memstore size.
> > >>>> 2012-10-16 00:00:00,358 INFO
> > >>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> > >>>> Opening of region {NAME =3D>
> > >>>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > >>>> STARTKEY =3D> '', ENDKEY =3D> '', ENCODED =3D>
> > >>>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as
> FAILED_OPEN in
> > >>>> ZK
> > >>>>
> > >>>> And we have a endpoint in base .After the base tried to load
> this
> > >>>> table ivy test_deu for 90,000 times ,the endpoint class also =
has
> > >> been
> > >>>> loaded for 90,000 times.
> > >>>> The jvm memory has been filled.
> > >>>> The gcutil shows
> > >>>> S0C    S1C    S0U    S1U      EC       EU        OC         OU
> > >> PC
> > >>>> PU    YGC     YGCT    FGC    FGCT     GCT
> > >>>> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34880.0  0.0   209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> > >>>> 34880.0 34880.0 34880.0  0.0   209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> > >>>>
> > >>>> The jmap dump file shows
> > >>>>
> > >>>> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
> > >>>> 191050 instances of class org.apache.hadoop.fs.Path
> > >>>> 187364 instances of class
> > >>>> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> > >>>> 187301 instances of class org.cliffc.high_scale_lib.Counter
> > >>>> 102272 instances of class
> > >> net.sf.ehcache.concurrent.ReadWriteLockSync
> > >>>> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
> > >>>> 93650 instances of class
> > >>>> com.google.common.collect.MutableClassToInstanceMap
> > >>>> 93650 instances of class DEUColumnAggregationEndpoint
> > >>>>
> > >>>> DEUColumnAggregationEndpoint is our endpoint class.
> > >>>>
> > >>>> We guess the 90,000 times check this table and load endpoint
> class
> > >>>> leads this memory leak.
> > >>>>
> > >>>> But how to drop this table?
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >
> > >
> >
> >
>=20
>=20
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera