accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russ Weeks <rwe...@newbrightidea.com>
Subject Re: Seeing lots of "Internal error processing waitForTableOperation"
Date Thu, 27 Feb 2014 21:34:25 GMT
Hi, Keith,

Thanks for your response.  I think that's exactly what happened. I screwed
up the config command and wound up specifying a couple of iterators on the
!METADATA table (and indeed all tables in the system).

I've removed those iterators using "config -d" and now I can scan the
!METADATA table without errors in the shell or the master log. But I'm
still not able to delete tables. When I run the command "deletetable
test_table", I see this message in the shell:

ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
processing waitForTableOperation

And this exception in the master log:

2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added
lock entry 0 userData 1a33c6891b997e71 lockType WRITE
2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n (1a33c6891b997e71)
locked for write operation: DELETE
2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition
table to state DELETING
2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
tid=1a33c6891b997e71
java.lang.RuntimeException:
org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
at
org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
at
org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
at
org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
at
org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:724)
Caused by:
org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
at
org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
at
org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
at com.sun.proxy.$Proxy10.mutate(Unknown Source)
at
org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
... 6 more
2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n (1a33c6891b997e71)
unlocked for write


-Russ




On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <keith@deenlo.com> wrote:

>
>
>
> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>
>> Hi, All,
>>
>> I can't delete any tables through the accumulo shell, I keep getting the
>> message: "Internal error processing waitForTableOperation".
>>
>> Recently my accumulo cluster has started showing this error repeatedly in
>> the master log:
>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
>> 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab
>> ~pr-� �}
>>
>> And this error in the monitor log:
>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
>> has no prev end row
>>
>> And this in the GC log:
>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no prev
>> end row
>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>
>> Maybe coincidence, but it seems like things started getting weird when I
>> added a WholeRowIterator through the accumulo shell (which is nonsense, I
>> know). The iterators on one of my tables (36M entries) look like this:
>>
>
> Was that iterator added at the table level or system level?  Adding at the
> system level could impact the metadata table, see ACCUMULO-324.
>
>
>>
>> -    Iterator WholeRowIterator, scan scope options:
>> -        iteratorPriority = 50
>> -        iteratorClassName =
>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>> -    Iterator rowdel, majc scope options:
>> -        iteratorPriority = 30
>> -        iteratorClassName =
>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>> -    Iterator transformer, majc scope options:
>> -        iteratorPriority = 15
>> -        iteratorClassName =
>> com.phemi.agile.datapolicy.DataColumnTransformer
>> -        maxBufferSize = 256M
>> -    Iterator vers, majc scope options:
>> -        iteratorPriority = 20
>> -        iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> -        maxVersions = 1
>> -    Iterator vers, minc scope options:
>> -        iteratorPriority = 20
>> -        iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> -        maxVersions = 1
>> -    Iterator vers, scan scope options:
>> -        iteratorPriority = 20
>> -        iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> -        maxVersions = 1
>>
>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>> WholeRowIterator. No errors were returned from the deleteiter command in
>> the shell but after running listiter, the WholeRowIterator and
>> RowDeletingIterator are still there.
>>
>> I *am* still able to create tables, but tables that I create can't then
>> be deleted.
>>
>> Any advice appreciated,
>> -Russ
>>
>>
>

Mime
View raw message