accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris McCubbin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2361) droptable created infinite METADATA scan loop
Date Tue, 18 Feb 2014 17:08:20 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904252#comment-13904252
] 

Chris McCubbin commented on ACCUMULO-2361:
------------------------------------------

Here's the output of some listscans on the monitor:

{code}
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
     10.10.1.107:9997 |    10.10.1.206:41386 |      3ms |        - |   IDLE | BATCH | !SYSTEM
|!METADATA |[loc::, chopped::, log::, ~tab:~pr:, future::] |                     |       N/A
|[wholeRows=1000,org.apache.accumulo.core.iterators.user.WholeRowIterator, tabletChange=1001,org.apache.accumulo.server.master.state.TabletStateChangeIterator]
| {wholeRows={}, tabletChange={tables=2,1,!0,5, merges=, servers=10.10.1.60:9997[144274cd3170007],10.10.1.209:9997[144274cd317000b],10.10.1.129:9997[144274cd317000a],10.10.1.114:9997[144274cd3170009],10.10.1.107:9997[144274cd3170008]}}
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
root@sqrrl> listscans
 TABLET SERVER        | CLIENT               | AGE      | LAST     | STATE  | TYPE  | USER
   | TABLE   | COLUMNS   | AUTHORIZATIONS      | TABLET    | ITERATORS  | ITERATOR OPTIONS
     10.10.1.209:9997 |    10.10.1.206:55270 |      1ms |        - |   IDLE | BATCH | !SYSTEM
|!METADATA |[loc::, chopped::, log::, ~tab:~pr:, future::] |                     |       N/A
|[wholeRows=1000,org.apache.accumulo.core.iterators.user.WholeRowIterator, tabletChange=1001,org.apache.accumulo.server.master.state.TabletStateChangeIterator]
| {wholeRows={}, tabletChange={tables=2,1,!0,5, merges=, servers=10.10.1.60:9997[144274cd3170007],10.10.1.209:9997[144274cd317000b],10.10.1.129:9997[144274cd317000a],10.10.1.114:9997[144274cd3170009],10.10.1.107:9997[144274cd3170008]}}
root@sqrrl> 
{code}

> droptable created infinite METADATA scan loop
> ---------------------------------------------
>
>                 Key: ACCUMULO-2361
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2361
>             Project: Accumulo
>          Issue Type: Bug
>    Affects Versions: 1.5.0
>            Reporter: Chris McCubbin
>            Assignee: Eric Newton
>         Attachments: Screen Shot 2014-02-12 at 2.27.11 PM.png, Screen Shot 2014-02-12
at 2.28.09 PM.png, jstack1.txt, jstack2.txt, jstack3.txt, masterJstack.txt, masterJstack2.txt,
masterJstack3.txt
>
>
> Working with [~vines] on this one...
> Setup: Created a couple tables, added some data, then dropped them. The drop hangs and
!METADATA (which has ~400 entries) is scanned in what looks like an infinite loop.
> The table being dropped loks like this in !METADATA:
> {code}
> root@sqrrl> scan -b 3 -e 5 -t !METADATA
> 4;\x00\x00\x06f srv:dir []    /t-00000b6
> 4;\x00\x00\x06f srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
> 4;\x00\x00\x06f srv:time []    M0
> 4;\x00\x00\x06f ~tab:~pr []    \x00
> 4;\x00\x00\x0C\xCC loc:144274cd3170008 []    10.10.1.107:9997
> 4;\x00\x00\x0C\xCC srv:dir []    /t-00000bj
> 4;\x00\x00\x0C\xCC srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
> 4;\x00\x00\x0C\xCC srv:time []    M0
> 4;\x00\x00\x0C\xCC ~tab:~pr []    \x01\x00\x00\x06f
> 4;\x00\x00\x133 srv:dir []    /t-000002h
> 4;\x00\x00\x133 srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
> 4;\x00\x00\x133 srv:time []    M0
> 4;\x00\x00\x133 ~tab:~pr []    \x01\x00\x00\x0C\xCC
> {code}
> We think this may be the relevant message in the master debug logs:
> {code}
> 2014-02-12 19:13:31,397 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,459 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,524 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,588 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,662 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,725 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,788 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,854 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> 2014-02-12 19:13:31,917 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted:
4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
> ...etc
> {code}
> Graceful accumulo reboot hangs. 
> Hard reboot of everything (control-c'd) clears the problem.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message