hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: How to delete a table manually
Date Fri, 21 Jan 2011 19:05:58 GMT
It seems there is a typo:
"we'll no interrupt the running compaction" should be "we'll now interrupt
the running compaction"

On Fri, Jan 21, 2011 at 10:47 AM, Stack <stack@duboce.net> wrote:

> On Fri, Jan 21, 2011 at 4:51 AM, Wayne <wav100@gmail.com> wrote:
> > After several hours I have figured out how to get the Disable command to
> > work and how to delete manually, but in the process there are 4 problems
> I
> > encountered that I think are areas that could be improved (or my
> > understanding improved).
> >
> > 1) The client timeout is used for the disable command which was my
> problem.
> > Does this totally make sense? Should a DML minded timeout be used for DDL
> > statements that we know can take a very long time normally with a large
> > cluster?
> >
>
> Sorry Wayne.  I meant to respond yesterday to your original query.
>
> Enable/Disable has been redone in 0.90.  Now there are added
> enabling/disabling states that are maintained up in zk and in shell
> there are commands is_enabled and is_disabled.  We still have the same
> (DML) timeout (sortof -- see below for more) but at least now if it
> times out, you are not hosed.  The disable or enable process is still
> running and you can query its state.  There is also notion of async
> enable/disable though this latter facility is not exposed in shell,
> only in the HBaseAdmin API.
>
>
> > 2) If the disable command fails the first time it does not "roll back".
> The
> > ONLY way to proceed is to enable and then try to disable again. The first
> > disable attempt is all that seems to work. Subsequent disable statements
> > usually work without errors but never seem to "work". The entire table
> > should be disabled after issuing this command or the entire table should
> > still be enabled. I was caught in this half disabled or mostly disabled
> > which was very frustating.
> >
>
> Sorry about that.   Should be better in 0.90.0.
>
> Things should run a bit faster in 0.90.0 too because disable used to
> include an update of .META. per region plus a close of all regions
> that make up the table.  In 0.90.0 there is no longer the .META.
> update and close is more prompt now; in the past close would wait on
> any running compactions to complete before proceeding.  In 0.90.0
> we'll no interrupt the running compaction so close happens the sooner.
>
> There is room for a bunch more improvement. For example, deleting a
> table, there should be short-circuit that punts on flush of in-memory
> state and clean-close of open regions.
>
> > 3) The biggest issue of all is why certain regions do not report back to
> the
> > disable command. What are the various states of a region that could cause
> > this? Compaction I know is one, what else could cause the disable command
> to
> > take too long? Shouldn't a disable force itself through and wait long
> enough
> > to be able to disable every region? Again a long wait time or a more
> > forceful operation would help.
> >
>
> It wasn't that smart in 0.20/0.89.  Its still pretty dumb but better in
> 0.90.0.
>
> Master process runs the enable/disable process in both old and new
> HBase.  In 0.20/0.89, it was a sync process w/ master waiting on
> regions to flip to 'offline' after successful close.  The state of
> disabledness was when all regions in table had 'offline' state.  Any
> hiccup, a problem closing or a failure to update .META. w/ offline per
> region would bork the disabling process.  It was super fragile.  We
> tried to talk it up as so.
>
> In 0.90, client queues in master an executor that flips table to
> disabling in zk and then in parallel sends out unassigns of all table
> regions.  The executor then hangs around with a more DDL-like timeout
> of hbase.bulk.assignment.waiton.empty.rit (10minutes by default).
> Meantime clients can check state of the disable.   After all unassigns
> complete, the table is flipped to disabled.
>
>
> > 4) Through all of the attempts to disable I saw regions coming and going
> and
> > nothing was consistent. The UI showed the table as disabled and listed 1
> > region in the table (there were 1000s). The node view listed several
> other
> > regions but not the same one as the table view. It was a very strange
> > situation. The UI to browse the tables and regions is great but it would
> be
> > even better if it gave a 100% view of regions and their current states. A
> > summary view of region counts per table based on state or status would be
> > fantastic.
>
> Please file a JIRA.  Sounds like good idea.  We could hoist stuff up
> out of hbck tool up into UI.
>
>
> > There is a compaction count, but what about in split, read/rite
> > lock, disabled, etc. What is the precise list of regions states that
> could
> > occur and show a summary count per state as well as detailed state for
> each
> > specific region in the list. Fundamentally this is the health monitor of
> the
> > system and as a dba I really need to know the 100% count of regions and
> > where they are all at in terms of availability. Are they disabled,
> blocked
> > for writes, blocked for reads, in compaction, etc. etc. If there are
> various
> > states that cause disabling to be blocked it can be reported here so that
> I
> > at least know when a disable command can be executed successfully (and
> this
> > should be documented).
> >
>
>
> Please file a JIRA.  This is great stuff.
>
> Sorry for pain caused messing w/ broke enable/disable.  It should be
> better in 0.90 and easier to fix if bugs.
>
> St.Ack
>
>
> > Thanks
> >
> > On Thu, Jan 20, 2011 at 9:01 PM, Wayne <wav100@gmail.com> wrote:
> >
> >> I need to delete some tables and I am not sure the best way to do it.
> The
> >> shell does not work. The disable command says it runs ok but every time
> I
> >> run drop or truncate I get an exception that says the table is not
> >> disabled.  The UI shows it as disabled but truncate/drop still do not
> work.
> >> I have even tried to restart the cluster as sometimes that makes the
> disable
> >> "stick".
> >>
> >> What is the best way to delete a table manually? My assumption is that
> with
> >> 10k regions in 3 tables that I need to delete that the shell is not
> going to
> >> work. How can I do this without a completely fresh install of
> everything?
> >> How can the data/tables be removed manually without too much pain?
> >>
> >> Thanks.
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message