accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russ Weeks <rwe...@newbrightidea.com>
Subject Re: Seeing lots of "Internal error processing waitForTableOperation"
Date Thu, 27 Feb 2014 21:50:17 GMT
Looking at Zookeeper, it seems like some of my tables are stuck in the
DELETING state. I can't delete these tables. I *am* able to create and
delete new tables, or delete other tables that are not in the DELETING
state.

Is there any way to force the table out of the DELETING state?

Thanks,
-Russ

[zk: localhost:2181(CONNECTED) 21] get
/accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state
DELETING
cZxid = 0x2001138c0
ctime = Wed Feb 26 20:08:17 PST 2014
mZxid = 0x2001138ef
mtime = Wed Feb 26 20:08:24 PST 2014
pZxid = 0x2001138c0
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 8
numChildren = 0
[zk: localhost:2181(CONNECTED) 22] get
/accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name
test_table
cZxid = 0x2001138bf
ctime = Wed Feb 26 20:08:17 PST 2014
mZxid = 0x2001138bf
mtime = Wed Feb 26 20:08:17 PST 2014
pZxid = 0x2001138bf
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 0




On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:

> Hi, Keith,
>
> Thanks for your response.  I think that's exactly what happened. I screwed
> up the config command and wound up specifying a couple of iterators on the
> !METADATA table (and indeed all tables in the system).
>
> I've removed those iterators using "config -d" and now I can scan the
> !METADATA table without errors in the shell or the master log. But I'm
> still not able to delete tables. When I run the command "deletetable
> test_table", I see this message in the shell:
>
> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
> processing waitForTableOperation
>
> And this exception in the master log:
>
> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added
> lock entry 0 userData 1a33c6891b997e71 lockType WRITE
> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n
> (1a33c6891b997e71) locked for write operation: DELETE
> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition
> table to state DELETING
> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
> tid=1a33c6891b997e71
> java.lang.RuntimeException:
> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>  at
> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
>  at
> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
> at
> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
>  at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
>  at java.lang.Thread.run(Thread.java:724)
> Caused by:
> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
> at
> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
>  at
> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
>  at
> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
> at com.sun.proxy.$Proxy10.mutate(Unknown Source)
>  at
> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
> ... 6 more
> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n
> (1a33c6891b997e71) unlocked for write
>
>
> -Russ
>
>
>
>
> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <keith@deenlo.com> wrote:
>
>>
>>
>>
>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>>
>>> Hi, All,
>>>
>>> I can't delete any tables through the accumulo shell, I keep getting the
>>> message: "Internal error processing waitForTableOperation".
>>>
>>> Recently my accumulo cluster has started showing this error repeatedly
>>> in the master log:
>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
>>> 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab
>>> ~pr-� �}
>>>
>>> And this error in the monitor log:
>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
>>> has no prev end row
>>>
>>> And this in the GC log:
>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>>> prev end row
>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>>
>>> Maybe coincidence, but it seems like things started getting weird when I
>>> added a WholeRowIterator through the accumulo shell (which is nonsense, I
>>> know). The iterators on one of my tables (36M entries) look like this:
>>>
>>
>> Was that iterator added at the table level or system level?  Adding at
>> the system level could impact the metadata table, see ACCUMULO-324.
>>
>>
>>>
>>> -    Iterator WholeRowIterator, scan scope options:
>>> -        iteratorPriority = 50
>>> -        iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>>> -    Iterator rowdel, majc scope options:
>>> -        iteratorPriority = 30
>>> -        iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>>> -    Iterator transformer, majc scope options:
>>> -        iteratorPriority = 15
>>> -        iteratorClassName =
>>> com.phemi.agile.datapolicy.DataColumnTransformer
>>> -        maxBufferSize = 256M
>>> -    Iterator vers, majc scope options:
>>> -        iteratorPriority = 20
>>> -        iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>> -        maxVersions = 1
>>> -    Iterator vers, minc scope options:
>>> -        iteratorPriority = 20
>>> -        iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>> -        maxVersions = 1
>>> -    Iterator vers, scan scope options:
>>> -        iteratorPriority = 20
>>> -        iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>> -        maxVersions = 1
>>>
>>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>>> WholeRowIterator. No errors were returned from the deleteiter command in
>>> the shell but after running listiter, the WholeRowIterator and
>>> RowDeletingIterator are still there.
>>>
>>> I *am* still able to create tables, but tables that I create can't then
>>> be deleted.
>>>
>>> Any advice appreciated,
>>> -Russ
>>>
>>>
>>
>

Mime
View raw message