Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4988D10CC1 for ; Tue, 18 Feb 2014 17:40:36 +0000 (UTC) Received: (qmail 74864 invoked by uid 500); 18 Feb 2014 17:40:28 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 74787 invoked by uid 500); 18 Feb 2014 17:40:26 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 74671 invoked by uid 99); 18 Feb 2014 17:40:24 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Feb 2014 17:40:24 +0000 Date: Tue, 18 Feb 2014 17:40:23 +0000 (UTC) From: "Eric Newton (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-2361) droptable created infinite METADATA scan loop MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904295#comment-13904295 ] Eric Newton commented on ACCUMULO-2361: --------------------------------------- Any chance you have the master debug logs from this? I see a shutdown is being requested in the master jstacks. Did you notice the problem and attempt a shutdown? Or did you request a shutdown after noticing the problem? > droptable created infinite METADATA scan loop > --------------------------------------------- > > Key: ACCUMULO-2361 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2361 > Project: Accumulo > Issue Type: Bug > Affects Versions: 1.5.0 > Reporter: Chris McCubbin > Assignee: Eric Newton > Attachments: Screen Shot 2014-02-12 at 2.27.11 PM.png, Screen Shot 2014-02-12 at 2.28.09 PM.png, jstack1.txt, jstack2.txt, jstack3.txt, masterJstack.txt, masterJstack2.txt, masterJstack3.txt > > > Working with [~vines] on this one... > Setup: Created a couple tables, added some data, then dropped them. The drop hangs and !METADATA (which has ~400 entries) is scanned in what looks like an infinite loop. > The table being dropped loks like this in !METADATA: > {code} > root@sqrrl> scan -b 3 -e 5 -t !METADATA > 4;\x00\x00\x06f srv:dir [] /t-00000b6 > 4;\x00\x00\x06f srv:lock [] tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b > 4;\x00\x00\x06f srv:time [] M0 > 4;\x00\x00\x06f ~tab:~pr [] \x00 > 4;\x00\x00\x0C\xCC loc:144274cd3170008 [] 10.10.1.107:9997 > 4;\x00\x00\x0C\xCC srv:dir [] /t-00000bj > 4;\x00\x00\x0C\xCC srv:lock [] tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b > 4;\x00\x00\x0C\xCC srv:time [] M0 > 4;\x00\x00\x0C\xCC ~tab:~pr [] \x01\x00\x00\x06f > 4;\x00\x00\x133 srv:dir [] /t-000002h > 4;\x00\x00\x133 srv:lock [] tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b > 4;\x00\x00\x133 srv:time [] M0 > 4;\x00\x00\x133 ~tab:~pr [] \x01\x00\x00\x0C\xCC > {code} > We think this may be the relevant message in the master debug logs: > {code} > 2014-02-12 19:13:31,397 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,459 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,524 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,588 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,662 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,725 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,788 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,854 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > 2014-02-12 19:13:31,917 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff > ...etc > {code} > Graceful accumulo reboot hangs. > Hard reboot of everything (control-c'd) clears the problem. -- This message was sent by Atlassian JIRA (v6.1.5#6160)