accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russ Weeks <rwe...@newbrightidea.com>
Subject Re: Seeing lots of "Internal error processing waitForTableOperation"
Date Thu, 27 Feb 2014 22:05:43 GMT
Hi, Keith,

hduser@agile01:~/accumulo/bin$ accumulo
org.apache.accumulo.server.fate.Admin print
txid: 254bf92a31b14e3b  status: SUCCESSFUL          op: TableRangeOp
locked: []              locking: []              top: null
txid: 32cedb4ad5b9b3e3  status: SUCCESSFUL          op: DeleteTable
 locked: []              locking: []              top: null

I might have jumped the gun a bit, but I found how to use the zookeeper CLI
to force the table state to ONLINE... I can delete my tables now, I hope I
haven't messed up the FSM at a higher level.

-Russ


On Thu, Feb 27, 2014 at 2:00 PM, Keith Turner <keith@deenlo.com> wrote:

> If you run 'accumulo org.apache.accumulo.server.fate.Admin -<list option,
> not sure what it is>', what does that look like?
>
>
>
>
> On Thu, Feb 27, 2014 at 4:50 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>
>> Looking at Zookeeper, it seems like some of my tables are stuck in the
>> DELETING state. I can't delete these tables. I *am* able to create and
>> delete new tables, or delete other tables that are not in the DELETING
>> state.
>>
>> Is there any way to force the table out of the DELETING state?
>>
>> Thanks,
>> -Russ
>>
>> [zk: localhost:2181(CONNECTED) 21] get
>> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state
>> DELETING
>> cZxid = 0x2001138c0
>> ctime = Wed Feb 26 20:08:17 PST 2014
>> mZxid = 0x2001138ef
>> mtime = Wed Feb 26 20:08:24 PST 2014
>> pZxid = 0x2001138c0
>> cversion = 0
>> dataVersion = 2
>> aclVersion = 0
>> ephemeralOwner = 0x0
>> dataLength = 8
>> numChildren = 0
>> [zk: localhost:2181(CONNECTED) 22] get
>> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name
>> test_table
>> cZxid = 0x2001138bf
>> ctime = Wed Feb 26 20:08:17 PST 2014
>> mZxid = 0x2001138bf
>> mtime = Wed Feb 26 20:08:17 PST 2014
>> pZxid = 0x2001138bf
>> cversion = 0
>> dataVersion = 0
>> aclVersion = 0
>> ephemeralOwner = 0x0
>> dataLength = 10
>> numChildren = 0
>>
>>
>>
>>
>> On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>>
>>> Hi, Keith,
>>>
>>> Thanks for your response.  I think that's exactly what happened. I
>>> screwed up the config command and wound up specifying a couple of iterators
>>> on the !METADATA table (and indeed all tables in the system).
>>>
>>> I've removed those iterators using "config -d" and now I can scan the
>>> !METADATA table without errors in the shell or the master log. But I'm
>>> still not able to delete tables. When I run the command "deletetable
>>> test_table", I see this message in the shell:
>>>
>>> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
>>> processing waitForTableOperation
>>>
>>> And this exception in the master log:
>>>
>>> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO :
>>> Added lock entry 0 userData 1a33c6891b997e71 lockType WRITE
>>> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n
>>> (1a33c6891b997e71) locked for write operation: DELETE
>>> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to
>>> transition table to state DELETING
>>> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
>>> tid=1a33c6891b997e71
>>> java.lang.RuntimeException:
>>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>>>  at
>>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
>>> at
>>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
>>>  at
>>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
>>> at
>>> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
>>>  at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
>>> at
>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>  at java.lang.Thread.run(Thread.java:724)
>>> Caused by:
>>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>>> at
>>> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
>>>  at
>>> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
>>> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>  at
>>> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
>>> at com.sun.proxy.$Proxy10.mutate(Unknown Source)
>>>  at
>>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
>>> ... 6 more
>>> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n
>>> (1a33c6891b997e71) unlocked for write
>>>
>>>
>>> -Russ
>>>
>>>
>>>
>>>
>>> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <keith@deenlo.com> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rweeks@newbrightidea.com>wrote:
>>>>
>>>>> Hi, All,
>>>>>
>>>>> I can't delete any tables through the accumulo shell, I keep getting
>>>>> the message: "Internal error processing waitForTableOperation".
>>>>>
>>>>> Recently my accumulo cluster has started showing this error repeatedly
>>>>> in the master log:
>>>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z<
:
>>>>> [] 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997~tab
~pr-� �}
>>>>>
>>>>> And this error in the monitor log:
>>>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet
>>>>> !0;!0< has no prev end row
>>>>>
>>>>> And this in the GC log:
>>>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has
no
>>>>> prev end row
>>>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>>>>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>>>>
>>>>> Maybe coincidence, but it seems like things started getting weird when
>>>>> I added a WholeRowIterator through the accumulo shell (which is nonsense,
I
>>>>> know). The iterators on one of my tables (36M entries) look like this:
>>>>>
>>>>
>>>> Was that iterator added at the table level or system level?  Adding at
>>>> the system level could impact the metadata table, see ACCUMULO-324.
>>>>
>>>>
>>>>>
>>>>> -    Iterator WholeRowIterator, scan scope options:
>>>>> -        iteratorPriority = 50
>>>>> -        iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>>>>> -    Iterator rowdel, majc scope options:
>>>>> -        iteratorPriority = 30
>>>>> -        iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>>>>> -    Iterator transformer, majc scope options:
>>>>> -        iteratorPriority = 15
>>>>> -        iteratorClassName =
>>>>> com.phemi.agile.datapolicy.DataColumnTransformer
>>>>> -        maxBufferSize = 256M
>>>>> -    Iterator vers, majc scope options:
>>>>> -        iteratorPriority = 20
>>>>> -        iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>>> -        maxVersions = 1
>>>>> -    Iterator vers, minc scope options:
>>>>> -        iteratorPriority = 20
>>>>> -        iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>>> -        maxVersions = 1
>>>>> -    Iterator vers, scan scope options:
>>>>> -        iteratorPriority = 20
>>>>> -        iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>>> -        maxVersions = 1
>>>>>
>>>>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>>>>> WholeRowIterator. No errors were returned from the deleteiter command
in
>>>>> the shell but after running listiter, the WholeRowIterator and
>>>>> RowDeletingIterator are still there.
>>>>>
>>>>> I *am* still able to create tables, but tables that I create can't
>>>>> then be deleted.
>>>>>
>>>>> Any advice appreciated,
>>>>> -Russ
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message