accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Seeing lots of "Internal error processing waitForTableOperation"
Date Thu, 27 Feb 2014 22:00:40 GMT
If you run 'accumulo org.apache.accumulo.server.fate.Admin -<list option,
not sure what it is>', what does that look like?




On Thu, Feb 27, 2014 at 4:50 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:

> Looking at Zookeeper, it seems like some of my tables are stuck in the
> DELETING state. I can't delete these tables. I *am* able to create and
> delete new tables, or delete other tables that are not in the DELETING
> state.
>
> Is there any way to force the table out of the DELETING state?
>
> Thanks,
> -Russ
>
> [zk: localhost:2181(CONNECTED) 21] get
> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state
> DELETING
> cZxid = 0x2001138c0
> ctime = Wed Feb 26 20:08:17 PST 2014
> mZxid = 0x2001138ef
> mtime = Wed Feb 26 20:08:24 PST 2014
> pZxid = 0x2001138c0
> cversion = 0
> dataVersion = 2
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 8
> numChildren = 0
> [zk: localhost:2181(CONNECTED) 22] get
> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name
> test_table
> cZxid = 0x2001138bf
> ctime = Wed Feb 26 20:08:17 PST 2014
> mZxid = 0x2001138bf
> mtime = Wed Feb 26 20:08:17 PST 2014
> pZxid = 0x2001138bf
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 10
> numChildren = 0
>
>
>
>
> On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>
>> Hi, Keith,
>>
>> Thanks for your response.  I think that's exactly what happened. I
>> screwed up the config command and wound up specifying a couple of iterators
>> on the !METADATA table (and indeed all tables in the system).
>>
>> I've removed those iterators using "config -d" and now I can scan the
>> !METADATA table without errors in the shell or the master log. But I'm
>> still not able to delete tables. When I run the command "deletetable
>> test_table", I see this message in the shell:
>>
>> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
>> processing waitForTableOperation
>>
>> And this exception in the master log:
>>
>> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added
>> lock entry 0 userData 1a33c6891b997e71 lockType WRITE
>> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n
>> (1a33c6891b997e71) locked for write operation: DELETE
>> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition
>> table to state DELETING
>> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
>> tid=1a33c6891b997e71
>> java.lang.RuntimeException:
>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>>  at
>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
>> at
>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
>>  at
>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
>> at
>> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
>>  at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
>> at
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>  at java.lang.Thread.run(Thread.java:724)
>> Caused by:
>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>> at
>> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
>>  at
>> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
>> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>  at
>> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
>> at com.sun.proxy.$Proxy10.mutate(Unknown Source)
>>  at
>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
>> ... 6 more
>> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n
>> (1a33c6891b997e71) unlocked for write
>>
>>
>> -Russ
>>
>>
>>
>>
>> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <keith@deenlo.com> wrote:
>>
>>>
>>>
>>>
>>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>>>
>>>> Hi, All,
>>>>
>>>> I can't delete any tables through the accumulo shell, I keep getting
>>>> the message: "Internal error processing waitForTableOperation".
>>>>
>>>> Recently my accumulo cluster has started showing this error repeatedly
>>>> in the master log:
>>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< :
>>>> [] 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997~tab
~pr-� �}
>>>>
>>>> And this error in the monitor log:
>>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet
>>>> !0;!0< has no prev end row
>>>>
>>>> And this in the GC log:
>>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>>>> prev end row
>>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>>>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>>>
>>>> Maybe coincidence, but it seems like things started getting weird when
>>>> I added a WholeRowIterator through the accumulo shell (which is nonsense,
I
>>>> know). The iterators on one of my tables (36M entries) look like this:
>>>>
>>>
>>> Was that iterator added at the table level or system level?  Adding at
>>> the system level could impact the metadata table, see ACCUMULO-324.
>>>
>>>
>>>>
>>>> -    Iterator WholeRowIterator, scan scope options:
>>>> -        iteratorPriority = 50
>>>> -        iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>>>> -    Iterator rowdel, majc scope options:
>>>> -        iteratorPriority = 30
>>>> -        iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>>>> -    Iterator transformer, majc scope options:
>>>> -        iteratorPriority = 15
>>>> -        iteratorClassName =
>>>> com.phemi.agile.datapolicy.DataColumnTransformer
>>>> -        maxBufferSize = 256M
>>>> -    Iterator vers, majc scope options:
>>>> -        iteratorPriority = 20
>>>> -        iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>> -        maxVersions = 1
>>>> -    Iterator vers, minc scope options:
>>>> -        iteratorPriority = 20
>>>> -        iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>> -        maxVersions = 1
>>>> -    Iterator vers, scan scope options:
>>>> -        iteratorPriority = 20
>>>> -        iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>> -        maxVersions = 1
>>>>
>>>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>>>> WholeRowIterator. No errors were returned from the deleteiter command in
>>>> the shell but after running listiter, the WholeRowIterator and
>>>> RowDeletingIterator are still there.
>>>>
>>>> I *am* still able to create tables, but tables that I create can't then
>>>> be deleted.
>>>>
>>>> Any advice appreciated,
>>>> -Russ
>>>>
>>>>
>>>
>>
>

Mime
View raw message