accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russ Weeks <rwe...@newbrightidea.com>
Subject Re: Seeing lots of "Internal error processing waitForTableOperation"
Date Thu, 27 Feb 2014 17:53:41 GMT
Hi, Josh,

No, I can't scan the !METADATA table. I get about 2k of output
(here<https://gist.github.com/rweeks/055f0f19c5b959af8afc>,
if useful) and then the shell blocks. I'm seeing a lot of these exceptions
in the monitor log:

Caused by:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164)
... 6 more

-Russ


On Wed, Feb 26, 2014 at 9:33 PM, Josh Elser <josh.elser@gmail.com> wrote:

> Hrm, looking at the code, it appears that the !METADATA table has some
> inconsistencies. A (at least?) one row lacks of the "prev-row" column.
> Something like:
>
> some_row ~tab:~pr
>
> Sadly, I'm not positive about the cause given the information. Can you
> scan all of the !METADATA table without error?
>
> That said, I don't think that the above would prevent you from being able
> to change the iterators on a table because that's stored purely in
> ZooKeeper. Maybe it tries to read some data from !METADATA before doing the
> ZooKeeper update?
>
>
> On 2/26/14, 11:49 PM, Russ Weeks wrote:
>
>> Hi, Josh,
>>
>> Thanks for responding so quickly... sorry, I should've mentioned that
>> I'm running Accumulo 1.5.0.
>>
>> Nothing really helpful that I can see in the monitor or debug monitor
>> logs. It just interleaves the 2 errors above from the
>> state.MetaDataTableScanner and util.TabletIterator loggers.
>>
>> -Russ
>>
>>
>> On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <josh.elser@gmail.com
>> <mailto:josh.elser@gmail.com>> wrote:
>>
>>     Hi Russ,
>>
>>     What version of Accumulo are you running?
>>
>>     Any other exceptions in the Monitor?
>>
>>     - Josh
>>
>>
>>     On 2/26/14, 11:10 PM, Russ Weeks wrote:
>>
>>         Hi, All,
>>
>>         I can't delete any tables through the accumulo shell, I keep
>>         getting the
>>         message: "Internal error processing waitForTableOperation".
>>
>>         Recently my accumulo cluster has started showing this error
>>         repeatedly
>>         in the master log:
>>         [state.MetaDataTableScanner] WARN : No prev-row for key extent:
>>         {z< : []
>>         9223372036854775807
>>         false=  loc243f8e8ad6f5875>. 10.128.1.17:9997
>>         <http://10.128.1.17:9997> ~tab ~pr-�  �}
>>
>>
>>         And this error in the monitor log:
>>         [util.TabletIterator] WARN : gc:agile01 Metadata problem :
>>         tablet !0;!0<
>>         has no prev end row
>>
>>         And this in the GC log:
>>         [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has
>> no
>>         prev end row
>>         [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 :
>> []
>>         9223372036854775807 false,~ : [] 9223372036854775807 false)
>>
>>         Maybe coincidence, but it seems like things started getting
>>         weird when I
>>         added a WholeRowIterator through the accumulo shell (which is
>>         nonsense,
>>         I know). The iterators on one of my tables (36M entries) look
>>         like this:
>>
>>         -    Iterator WholeRowIterator, scan scope options:
>>         -        iteratorPriority = 50
>>         -        iteratorClassName =
>>         org.apache.accumulo.core.__iterators.user.__WholeRowIterator
>>
>>         -    Iterator rowdel, majc scope options:
>>         -        iteratorPriority = 30
>>         -        iteratorClassName =
>>         org.apache.accumulo.core.__iterators.user.__RowDeletingIterator
>>
>>         -    Iterator transformer, majc scope options:
>>         -        iteratorPriority = 15
>>         -        iteratorClassName =
>>         com.phemi.agile.datapolicy.__DataColumnTransformer
>>
>>         -        maxBufferSize = 256M
>>         -    Iterator vers, majc scope options:
>>         -        iteratorPriority = 20
>>         -        iteratorClassName =
>>         org.apache.accumulo.core.__iterators.user.__VersioningIterator
>>
>>         -        maxVersions = 1
>>         -    Iterator vers, minc scope options:
>>         -        iteratorPriority = 20
>>         -        iteratorClassName =
>>         org.apache.accumulo.core.__iterators.user.__VersioningIterator
>>
>>         -        maxVersions = 1
>>         -    Iterator vers, scan scope options:
>>         -        iteratorPriority = 20
>>         -        iteratorClassName =
>>         org.apache.accumulo.core.__iterators.user.__VersioningIterator
>>
>>         -        maxVersions = 1
>>
>>         I tried deleting DataColumnTransformer, RowDeletingIterator and
>>         WholeRowIterator. No errors were returned from the deleteiter
>>         command in
>>         the shell but after running listiter, the WholeRowIterator and
>>         RowDeletingIterator are still there.
>>
>>         I *am* still able to create tables, but tables that I create
>>         can't then
>>         be deleted.
>>
>>         Any advice appreciated,
>>         -Russ
>>
>>
>>

Mime
View raw message