Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 907D3D572 for ; Thu, 23 May 2013 00:05:44 +0000 (UTC) Received: (qmail 79949 invoked by uid 500); 23 May 2013 00:05:44 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 79909 invoked by uid 500); 23 May 2013 00:05:44 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 79900 invoked by uid 99); 23 May 2013 00:05:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 May 2013 00:05:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of david.medinets@gmail.com designates 209.85.219.51 as permitted sender) Received: from [209.85.219.51] (HELO mail-oa0-f51.google.com) (209.85.219.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 May 2013 00:05:38 +0000 Received: by mail-oa0-f51.google.com with SMTP id f4so3583676oah.38 for ; Wed, 22 May 2013 17:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=G+ZcOy/QgCkKzUDy84TCZno+gYwbxc+kU+Yei/B4aTg=; b=B/9ANPM+9ygL3cGlCzhJrnLGcT8XVe8sjuOWOgJlGeG3f8iwxn6PfR0iloB1xvyqlw mZmlIF0kT3SmDl2qhuvX6mtZmAR1bWWUf/Au5gcrzMumuuASTo9QGS9Nc1fR9BSfX7+D zjrD/xY8jTPv7xXOP7SypjrHvMBpqIby0Wikqw3bLubvT+WyH3ajEOIkyViaWZBNOJZb HGvbUGXDpt39v2YRnyqOmi+nT60jo6lKczFH6hgGdEKTxZTbEp2nDccqv4I8PsqjwDW5 byNTHe9O5gGdZEfZrb0pSNyNUi7nZJdQq02WPHjSSB6/8HwmawhnYC7R1ZktzZRw5g0U 3mvw== MIME-Version: 1.0 X-Received: by 10.182.215.130 with SMTP id oi2mr6537388obc.97.1369267517231; Wed, 22 May 2013 17:05:17 -0700 (PDT) Received: by 10.60.103.194 with HTTP; Wed, 22 May 2013 17:05:17 -0700 (PDT) Received: by 10.60.103.194 with HTTP; Wed, 22 May 2013 17:05:17 -0700 (PDT) In-Reply-To: References: Date: Wed, 22 May 2013 20:05:17 -0400 Message-ID: Subject: Re: Any way to recover from a corrupt block on the metadata table? From: David Medinets To: accumulo-user Content-Type: multipart/alternative; boundary=001a11c2b062f7998304dd576f47 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2b062f7998304dd576f47 Content-Type: text/plain; charset=ISO-8859-1 Has this procedure made it into the user docs? Seems useful. On May 22, 2013 3:27 PM, "Eric Newton" wrote: > Yes, but you're not going to like the answer. :-) > > Here's the high-level recipe: > > 1) record the table ids for your tables > 2) kill all the accumulo servers > 3) move /accumulo to a backup location > 4) re-initialize, recreate your tables and users > 5) use "importDirectory" to load the files in your backup into your new > tables > > You will want to script this last part. > > -Eric > > > > > On Wed, May 22, 2013 at 2:54 PM, Mike Hugo wrote: > >> We filled the disk on a test server (single node) and looks like we >> corrupted some files in the DFS. In particular, the metadata table is >> having some issues >> >> Accumulo is reporting: >> >> exception trying to assign tablet !0;!0<< /root_tablet >> java.io.IOException: Could not obtain block: >> blk_7026126848942509929_17401 >> file=/accumulo/tables/!0/root_tablet/A0000ct9.rf >> >> And hadoop fsck is showing: >> >> >> >> /accumulo/tables/!0/default_tablet/A0000ctb.rf 1303 bytes, 1 block(s): >> >> >> /accumulo/tables/!0/default_tablet/A0000ctb.rf: CORRUPT block blk_8698622187813164150 >> >> MISSING 1 blocks of total size 1303 B >> >> 0. blk_8698622187813164150_17402 len=1303 MISSING! >> >> >> /accumulo/tables/!0/root_tablet >> >> /accumulo/tables/!0/root_tablet/A0000ct9.rf 705 bytes, 1 block(s): >> >> >> /accumulo/tables/!0/root_tablet/A0000ct9.rf: CORRUPT block blk_7026126848942509929 >> >> MISSING 1 blocks of total size 705 B >> >> 0. blk_7026126848942509929_17401 len=705 MISSING! >> >> >> /accumulo/tables/!0/table_info >> >> /accumulo/tables/!0/table_info/A0000cta.rf 37857 bytes, 1 block(s): >> >> >> /accumulo/tables/!0/table_info/A0000cta.rf: CORRUPT block blk_8020296141595499911 >> >> MISSING 1 blocks of total size 37857 B >> >> 0. blk_8020296141595499911_17401 len=37857 MISSING! >> >> >> >> Is there a way to recover from this? >> >> >> >> >> Thanks, >> >> >> >> Mike >> >> > --001a11c2b062f7998304dd576f47 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Has this procedure made it into the user docs? Seems useful.

On May 22, 2013 3:27 PM, "Eric Newton"= <eric.newton@gmail.com>= wrote:
Yes, but you're not going to like the answer. :-)
=
Here's the high-level recipe:

1= ) record the table ids for your tables
2) kill all the accumulo s= ervers
3) move /accumulo to a backup location
4) re-initialize, rec= reate your tables and users
5) use "importDirectory" = =A0to load the files in your backup into your new tables

You will want to script this last part.

<= /div>
-Eric




On Wed, May 22, 2013 at 2:54 PM, Mike Hugo <mike@piragua.com>= wrote:
We filled the disk on a test server (single node) and look= s like we corrupted some files in the DFS. =A0In particular, the metadata t= able is having some issues

Accumulo is reporting:

exception trying to assign tablet !0;!0<< /r= oot_tablet
java.io.I= OException: Could not obtain block: blk_7026126848942509929_17401 file=3D/a= ccumulo/tables/!0/root_tablet/A0000ct9.rf

And hadoop fsck is showing:=





/accumulo/tables/!0/default_tablet/A0000ctb.rf=
 1303 bytes, 1 block(s):


/accumulo/tables/!0=
/default_tablet/A0000ctb.rf: CORRUPT block blk_8698622187813164150
=A0MISSING 1 blocks of total size 1=
303 B





0. blk_8698622187813164150_17402 len=3D1303 MI=
SSING!

<=
/pre>
<=
span style=3D"white-space:normal">/accumulo/tables/!0/root_tablet <dir&g=
t;





/accumulo/tables/!0/root_tablet/A0000ct9.rf 70=
5 bytes, 1 block(s):


/accumulo/tables/!0=
/root_tablet/A0000ct9.rf: CORRUPT block blk_7026126848942509929
=A0MISSING 1 blocks of total size 705 =
B





0. blk_7026126848942509929_17401 len=3D705 MIS=
SING!

<=
/pre>
<=
span style=3D"white-space:normal">/accumulo/tables/!0/table_info <dir>=
;





/accumulo/tables/!0/table_info/A0000cta.rf 378=
57 bytes, 1 block(s):


/accumulo/tables/!0=
/table_info/A0000cta.rf: CORRUPT block blk_8020296141595499911
=A0MISSING 1 blocks of total size 37857=
 B





0. blk_8020296141595499911_17401 len=3D37857 M=
ISSING!



Is there a way to reco=
ver from this?








Thanks,




Mike

--001a11c2b062f7998304dd576f47--