accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Seeing lots of "Internal error processing waitForTableOperation"
Date Thu, 27 Feb 2014 05:33:04 GMT
Hrm, looking at the code, it appears that the !METADATA table has some 
inconsistencies. A (at least?) one row lacks of the "prev-row" column. 
Something like:

some_row ~tab:~pr

Sadly, I'm not positive about the cause given the information. Can you 
scan all of the !METADATA table without error?

That said, I don't think that the above would prevent you from being 
able to change the iterators on a table because that's stored purely in 
ZooKeeper. Maybe it tries to read some data from !METADATA before doing 
the ZooKeeper update?

On 2/26/14, 11:49 PM, Russ Weeks wrote:
> Hi, Josh,
>
> Thanks for responding so quickly... sorry, I should've mentioned that
> I'm running Accumulo 1.5.0.
>
> Nothing really helpful that I can see in the monitor or debug monitor
> logs. It just interleaves the 2 errors above from the
> state.MetaDataTableScanner and util.TabletIterator loggers.
>
> -Russ
>
>
> On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com>> wrote:
>
>     Hi Russ,
>
>     What version of Accumulo are you running?
>
>     Any other exceptions in the Monitor?
>
>     - Josh
>
>
>     On 2/26/14, 11:10 PM, Russ Weeks wrote:
>
>         Hi, All,
>
>         I can't delete any tables through the accumulo shell, I keep
>         getting the
>         message: "Internal error processing waitForTableOperation".
>
>         Recently my accumulo cluster has started showing this error
>         repeatedly
>         in the master log:
>         [state.MetaDataTableScanner] WARN : No prev-row for key extent:
>         {z< : []
>         9223372036854775807
>         false=  loc243f8e8ad6f5875>. 10.128.1.17:9997
>         <http://10.128.1.17:9997> ~tab ~pr-�  �}
>
>         And this error in the monitor log:
>         [util.TabletIterator] WARN : gc:agile01 Metadata problem :
>         tablet !0;!0<
>         has no prev end row
>
>         And this in the GC log:
>         [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>         prev end row
>         [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>         9223372036854775807 false,~ : [] 9223372036854775807 false)
>
>         Maybe coincidence, but it seems like things started getting
>         weird when I
>         added a WholeRowIterator through the accumulo shell (which is
>         nonsense,
>         I know). The iterators on one of my tables (36M entries) look
>         like this:
>
>         -    Iterator WholeRowIterator, scan scope options:
>         -        iteratorPriority = 50
>         -        iteratorClassName =
>         org.apache.accumulo.core.__iterators.user.__WholeRowIterator
>         -    Iterator rowdel, majc scope options:
>         -        iteratorPriority = 30
>         -        iteratorClassName =
>         org.apache.accumulo.core.__iterators.user.__RowDeletingIterator
>         -    Iterator transformer, majc scope options:
>         -        iteratorPriority = 15
>         -        iteratorClassName =
>         com.phemi.agile.datapolicy.__DataColumnTransformer
>         -        maxBufferSize = 256M
>         -    Iterator vers, majc scope options:
>         -        iteratorPriority = 20
>         -        iteratorClassName =
>         org.apache.accumulo.core.__iterators.user.__VersioningIterator
>         -        maxVersions = 1
>         -    Iterator vers, minc scope options:
>         -        iteratorPriority = 20
>         -        iteratorClassName =
>         org.apache.accumulo.core.__iterators.user.__VersioningIterator
>         -        maxVersions = 1
>         -    Iterator vers, scan scope options:
>         -        iteratorPriority = 20
>         -        iteratorClassName =
>         org.apache.accumulo.core.__iterators.user.__VersioningIterator
>         -        maxVersions = 1
>
>         I tried deleting DataColumnTransformer, RowDeletingIterator and
>         WholeRowIterator. No errors were returned from the deleteiter
>         command in
>         the shell but after running listiter, the WholeRowIterator and
>         RowDeletingIterator are still there.
>
>         I *am* still able to create tables, but tables that I create
>         can't then
>         be deleted.
>
>         Any advice appreciated,
>         -Russ
>
>

Mime
View raw message