accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: How can I remove or fix a hanging droptable in Accumulo?
Date Thu, 13 Oct 2016 11:38:02 GMT
Jeff,

This is the bug I ran into,
https://issues.apache.org/jira/browse/ACCUMULO-4143.  Looks like it was
around migrations instead of splitting.

Mike

On Wed, Oct 12, 2016 at 8:27 PM, Michael Wall <mjwall@gmail.com> wrote:

> Jeff,
>
> There was a race condition around splitting I think, but I can't find the
> issue right now.  I caught a stacktrace at one point doing just as Ed
> suggests, then setting 'debug on' in the shell try the droptable again.
> After I found the issue, I turned off all ingest and waited for the master
> stop balancing and tables to stop splitting.  My droptable was successfully
> issued after that (doing Ed's trick once again).
>
> I can look for that issue tomorrow or if you catch a stacktrace, post it
> here.
>
> Mike
>
> On Wed, Oct 12, 2016 at 7:57 PM, Ed Coleman <dev1@etcoleman.com> wrote:
>
>> This is a brute force approach and should only done as a last resort.
>> Sometimes bouncing the master(s) helps – make sure all of stopped, not just
>> failed over to another instance., but it sounds like you have tried that.
>>
>>
>>
>> You may be able to force it by manually editing zookeeper  nodes using
>> zkCli.sh You are editing zookeeper so make sure you have the correct table
>> id and path(s)
>>
>>
>>
>> Add permission to write in zookeeper using:
>>
>>
>>
>> zkCli> addauth digest accumulo:YOUR_SECRET
>>
>>
>>
>> Under /accumulo/INSTANCE_ID/tables/TABLE_ID/state (going from memory,
>> could be fate) you can use examine the state
>>
>>
>>
>> zkCli> ls /accumulo/INSTANCE_ID/tables/TABLE_ID/
>>
>>
>>
>> there should be mutlple zk nodes, one of which is state (or maybe fate?)
>>
>>
>>
>> zkCli> get /accumulo/INSTANCE_ID/tables/TABLE_ID/state
>>
>>
>>
>> – it should say DELETING.
>>
>>
>>
>> zkCli> set /accumulo/INSTANCE_ID/tables/TABLE_ID/state  ONLINE
>>
>>
>>
>> This will force the table back online. After all tablets are accounted
>> for in the monitor you should be able to re-issue the delete command via
>> the accumulo shell.
>>
>>
>>
>> Ed Coleman
>>
>>
>>
>> *From:* Jeff Kubina [mailto:jeff.kubina@gmail.com]
>> *Sent:* Wednesday, October 12, 2016 7:38 PM
>> *To:* user@accumulo.apache.org
>> *Subject:* Re: How can I remove or fix a hanging droptable in Accumulo?
>>
>>
>>
>> Lots of bulk ingest assignments. It's been hung for days and even
>> returned after a reboot of the master and tservers.
>>
>>
>> --
>>
>> Jeff Kubina
>> 410-988-4436
>>
>>
>>
>>
>>
>> On Wed, Oct 12, 2016 at 6:28 PM, Michael Wall <mjwall@gmail.com> wrote:
>>
>> Jeff,
>>
>> What is the master doing while the droptable is hung?
>>
>> Mike
>>
>>
>>
>> On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <jeff.kubina@gmail.com>
>> wrote:
>>
>> How can I remove or fix a hanging droptable in Accumulo?
>>
>>
>>
>>
>>
>
>

Mime
View raw message