Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1FC04EC21 for ; Sun, 30 Dec 2012 18:59:39 +0000 (UTC) Received: (qmail 85536 invoked by uid 500); 30 Dec 2012 18:59:37 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 85462 invoked by uid 500); 30 Dec 2012 18:59:36 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 85454 invoked by uid 99); 30 Dec 2012 18:59:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Dec 2012 18:59:36 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.214.170] (HELO mail-ob0-f170.google.com) (209.85.214.170) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Dec 2012 18:59:31 +0000 Received: by mail-ob0-f170.google.com with SMTP id wp18so11148058obc.1 for ; Sun, 30 Dec 2012 10:59:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=+AbKp8SE8A5/1gt5BFRZGNROyzLUOMDkZGq7IEVuSf0=; b=AUp22JcOfcQd+B4aZ4l+qPJAPUZeZpznXwCM36Q//MhF35LtxVh8FDdIuX42HdOdN3 TxaoozDHO+VrQjWCm5+YA8tLLD+2bEyL5m3la7i5+8ezMMQHnqnRCawLAFz2yiJBIHK+ OwQXHh8LdIAy0OHou2kRIhd8t1xd1rGFTJy574PmxtyM/nl7avu/eC+oplUXz+uVWNjm 57A3LNQsyQiMWRONBylZUnwxB22e1QnmTFi3ftelge9HW7VJZoLb+a4JpRAsSXMqFihQ EwzmaH5ELRxWCLwQgVIak4Yo8QJH+cMIR7npmsMbu1Qmg/5WAA+ENM/boaox7N0eT2pL AzkQ== MIME-Version: 1.0 Received: by 10.182.88.3 with SMTP id bc3mr32079539obb.8.1356893949614; Sun, 30 Dec 2012 10:59:09 -0800 (PST) Received: by 10.182.153.36 with HTTP; Sun, 30 Dec 2012 10:59:09 -0800 (PST) In-Reply-To: References: Date: Sun, 30 Dec 2012 13:59:09 -0500 Message-ID: Subject: Re: CleanerChore exception From: Jean-Marc Spaggiari To: user@hbase.apache.org Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQljYM3VtfhoFviwMmq0Fc/kLo0oH7Ebr2TcRP5yqLBuluss8QyTcWuC7LQkPS1BCdVAR8yS X-Virus-Checked: Checked by ClamAV on apache.org So. Looking deeper I found few things. First, why checkAndDeleteDirectory is not "simply" calling FSUtils.delete (fs, toCheck, true)? I guess it's doing the same thing? Also, FSUtils.listStatus(fs, toCheck, null); will return null if there is no status. Not just an empty array. And it's returning null, we will exit without calling the delete methode. I tried to manually create a file on one of those directories. The exception disapears for 300 seconds because of the TTL for the newly created file. After 300 seconds, the file I pushed AND the directory got removed. So the issue is really with empty directories. I will take a look at what is in the trunk and in 0.94.4 to see if it's the same issue. But I think we can simple change all this code by a call to FSUtils.delete. I can open a JIRA and submit a patch for that. Just let me know. JM 2012/12/30, Jean-Marc Spaggiari : > Regargind the logcleaner settings, I have not changed anything. It's > what came with the initial install. So I don't have anything setup for > this plugin in my configuration files. > > For the files on the FS, here is what I have: > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls > /hbase/.archive/entry_duplicate > Found 30 items > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/00c185bc44b6dcf85a90b83bdda4ec2e > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/0ddf0d1802c6afd97d032fd09ea9e37d > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/18cf7c5c946ddf33e49b227feedfb688 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/2353f10e79dacc5cf201be6a1eb63607 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > /hbase/.archive/entry_duplicate/243f4007cf05415062010a5650598bff > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > /hbase/.archive/entry_duplicate/287682333698e36cea1670f5479fbf18 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/3742da9bd798342e638e1ce341f27537 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > /hbase/.archive/entry_duplicate/435c9c08bc08ed7248a013b6ffaa163b > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/45346b4b4248d77d45e031ea71a1fb63 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/4afe48fe6d8defe569f8632dd2514b07 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/68a4e364fe791a0d1f47febbb41e8112 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/7673d718962535c7b54cef51830f22a5 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > /hbase/.archive/entry_duplicate/7df6845ae9d052f4eae4a01e39313d61 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/8c5a263167d1b09f645af8efb4545554 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/8c98d9c635ba30d467d127a2ec1c69f8 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/8dfa96393e18ecca826fd9200e6bf68b > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/8e8f532e91a7197cd53b7626130be698 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/8eca1a325fe442a8546e43ac2f00cfef > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/9ad4c0551b90ea7717d7e3aaec76dc26 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/a135ccbc6f61ce544dbd537dc12489e9 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/a3d0332a6d51a8b15b99d1caca3f355a > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/bd2b8c942af27e541e20e430d506d2c0 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/c10c3a66948bde75fc41349108d86cf9 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > /hbase/.archive/entry_duplicate/cbf2f178691bfca8a7e9825115629b8e > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/d14a2546eaceede73b282e444ad1bb40 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > /hbase/.archive/entry_duplicate/d570a21a39e04ba2ec896bbe7166423c > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/e943bda56acd6beb35bdd56f0560f87f > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/ef5692ba83aba48d9e7a6b9c2cd0661e > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/fd85dd319c289959a790faed32ef1530 > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > /hbase/.archive/entry_duplicate/ffcdf6554accda1800e74838b67d3004 > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls > /hbase/.archive/entry_duplicate/00c185bc44b6dcf85a90b83bdda4ec2e > hadoop@node3:~/hadoop-1.0.3$ > > I have not lookeqd into ALL the subdirectories, but the 10 first are > empty. > > I see that there is some traces on checkAndDeleteDirectory... I will > try to activate that and see if there is more details. > > > JM > > 2012/12/30, Ted Yu : >> The exception came from this line: >> if (file.isDir()) checkAndDeleteDirectory(file.getPath()); >> Looking at checkAndDeleteDirectory(), it recursively deletes files and >> directories under the specified path. >> >> Does /hbase/.archive/entry_duplicate only contain empty directories >> underneath it ? >> >> You didn't modify the logcleaner plugin setting, right ? >> >> hbase.master.logcleaner.plugins >> >> org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner >> >> >> >> Cheers >> >> On Sun, Dec 30, 2012 at 9:53 AM, Jean-Marc Spaggiari < >> jean-marc@spaggiari.org> wrote: >> >>> I was going to move to 0.94.4 today ;) And yes I'm using 0.94.3. I >>> might wait a bit in case some testing is required with my version. >>> >>> Is this what you are looking for? http://pastebin.com/N8Q0FMba >>> >>> I will keep the files for now since it seems it's not causing any >>> major issue. That will allow some more testing if required. >>> >>> JM >>> >>> >>> 2012/12/30, Ted Yu : >>> > Looks like you're using 0.94.3 >>> > >>> > The archiver is backport of: >>> > HBASE-5547, Don't delete HFiles in backup mode >>> > >>> > Can you provide more the log where the IOE was reported using pastebin >>> > ? >>> > >>> > Thanks >>> > >>> > On Sun, Dec 30, 2012 at 9:08 AM, Jean-Marc Spaggiari < >>> > jean-marc@spaggiari.org> wrote: >>> > >>> >> Hi, >>> >> >>> >> I have a "IOException" /hbase/.archive/table_name is non empty >>> >> exception every minute on my logs. >>> >> >>> >> There is 30 directories under this directory. the main directory is >>> >> from yesterday, but all sub directories are from December 10th, all >>> >> the same time. >>> >> >>> >> What does this .archive directory is used for, and what should I do? >>> >> >>> >> Thanks, >>> >> >>> >> JM >>> >> >>> > >>> >> >