Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CADcMMgEG9yx1VCMeFGAPM4qWvpu63XBkys6pL-kyKoZvPfu_hA@mail.gmail.com>
References: 
 <CAB+7sRh2ynUUixD6OxgC8bGbRHA8PXbjSw+AoXxJb8QrK6YtUw@mail.gmail.com>
 <CADcMMgEG9yx1VCMeFGAPM4qWvpu63XBkys6pL-kyKoZvPfu_hA@mail.gmail.com>
From: Andrew Purtell <apurtell@apache.org>
Date: Wed, 18 Mar 2015 11:41:25 -0700
Message-ID: 
 <CA+RK=_Anb92pJ6U7dNZ1P+ntOPXZX1rQJi-NU=Osq93XQ8HX8g@mail.gmail.com>
Subject: Re: Recovering from corrupt blocks in HFile
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11359ede9ed427051194701c

--001a11359ede9ed427051194701c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

=E2=80=8B
On Tue, Mar 17, 2015 at 9:47 PM, Stack <stack@duboce.net> wrote:
>
> > If it's possible to recover all of the file except
> > a portion of the affected block, that would be OK too.
>
> I actually do not see a 'fix' or 'recover' on the hfile tool. We need to
> add it so you can recover all but the bad block (we should figure how to
> skip the bad section also).


=E2=80=8BI was just getting caught up on this thread and had the same thoug=
ht. Is
there an issue filed for this?


On Tue, Mar 17, 2015 at 9:47 PM, Stack <stack@duboce.net> wrote:

> On Tue, Mar 17, 2015 at 5:04 PM, Mike Dillon <mike.dillon@synctree.com>
> wrote:
>
> > Hi all-
> >
> > I've got an HFile that's reporting a corrupt block in "hadoop fsck" and
> was
> > hoping to get some advice on recovering as much data as possible.
> >
> > When I examined the blk-* file on the three data nodes that have a
> replica
> > of the affected block, I saw that the replicas on two of the datanodes
> had
> > the same SHA-1 checksum and that the replica on the other datanode was =
a
> > truncated version of the replica found on the other nodes (as reported
> by a
> > difference at EOF by "cmp"). The size of the two identical blocks is
> > 67108864, the same as most of the other blocks in the file.
> >
> > Given that there were two datanodes with the same data and another with
> > truncated data, I made a backup of the truncated file and dropped the
> > full-length copy of the block in its place directly on the data mount,
> > hoping that this would cause HDFS to no longer report the file as
> corrupt.
> > Unfortunately, this didn't seem to have any effect.
> >
> >
> That seems like a reasonable thing to do.
>
> Did you restart the DN that was serving this block before you ran fsck?
> (Fsck asks namenode what blocks are bad; it likely is still reporting off
> old info).
>
>
>
> > Looking through the Hadoop source code, it looks like there is a
> > CorruptReplicasMap internally that tracks which nodes have "corrupt"
> copies
> > of a block. In HDFS-6663 <
> https://issues.apache.org/jira/browse/HDFS-6663
> > >,
> > a "-blockId" parameter was added to "hadoop fsck" to allow dumping the
> > reason that a block ids is considered corrupt, but that wasn't added
> until
> > Hadoop 2.7.0 and our client is running 2.0.0-cdh4.6.0.
> >
> >
> Good digging.
>
>
>
> > I also had a look at running the "HFile" tool on the affected file (cf.
> > section 9.7.5.2.2 at http://hbase.apache.org/0.94/book/regions.arch.htm=
l
> ).
> > When I did that, I was able to see the data up to the corrupted block a=
s
> > far as I could tell, but then it started repeatedly looping back to the
> > first row and starting over. I believe this is related to the behavior
> > described in https://issues.apache.org/jira/browse/HBASE-12949
>
>
>
> So, your file is 3G and your blocks are 128M?
>
> The dfsclient should just pass over the bad replica and move on to the go=
od
> one so it would seem to indicate all replicas are bad for you.
>
> If you enable DFSClient DEBUG level logging it should report which blocks
> it is reading from. For example, here I am reading the start of the index
> blocks with DFSClient DEBUG enabled but I grep out the DFSClient emission=
s
> only:
>
> [stack@c2020 ~]$ ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.io.hfile.HFile -h -f
>
> /hbase/data/default/tsdb/3f4ea5ea14653cee6006f13c7d06d10b/t/68b00cb158aa4=
d839f1744639880f362|grep
> DFSClient
> 2015-03-17 21:42:56,950 DEBUG [main] util.ChecksumType:
> org.apache.hadoop.util.PureJavaCrc32 available
> 2015-03-17 21:42:56,952 DEBUG [main] util.ChecksumType:
> org.apache.hadoop.util.PureJavaCrc32C available
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/home/stack/hbase-1.0.1-SNAPSHOT/lib/slf4j-log4j12-1.7.7.jar!/o=
rg/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/slf4j=
-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2015-03-17 21:42:58,082 INFO  [main] hfile.CacheConfig:
> CacheConfig:disabled
> 2015-03-17 21:42:58,126 DEBUG [main] hdfs.DFSClient: newInfo =3D
> LocatedBlocks{
>   fileLength=3D108633903
>   underConstruction=3Dfalse
>
>
> blocks=3D[LocatedBlock{BP-410607956-10.20.84.26-1391491814882:blk_1078238=
905_1099516142201;
> getBlockSize()=3D108633903; corrupt=3Dfalse; offset=3D0;
> locs=3D[DatanodeInfoWithStorage[10.20.84.27:50011
> ,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK],
> DatanodeInfoWithStorage[10.20.84.31:50011
> ,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK],
> DatanodeInfoWithStorage[10.20.84.30:50011
> ,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]]}]
>
>
> lastLocatedBlock=3DLocatedBlock{BP-410607956-10.20.84.26-1391491814882:bl=
k_1078238905_1099516142201;
> getBlockSize()=3D108633903; corrupt=3Dfalse; offset=3D0;
> locs=3D[DatanodeInfoWithStorage[10.20.84.30:50011
> ,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK],
> DatanodeInfoWithStorage[10.20.84.31:50011
> ,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK],
> DatanodeInfoWithStorage[10.20.84.27:50011
> ,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]]}
>   isLastBlockComplete=3Dtrue}
> 2015-03-17 21:42:58,132 DEBUG [main] hdfs.DFSClient: Connecting to datano=
de
> 10.20.84.27:50011
> 2015-03-17 21:42:58,281 DEBUG [main] hdfs.DFSClient: Connecting to datano=
de
> 10.20.84.27:50011
> 2015-03-17 21:42:58,375 DEBUG [main] hdfs.DFSClient: newInfo =3D
> LocatedBlocks{
>   fileLength=3D108633903
>   underConstruction=3Dfalse
>
>
> blocks=3D[LocatedBlock{BP-410607956-10.20.84.26-1391491814882:blk_1078238=
905_1099516142201;
> getBlockSize()=3D108633903; corrupt=3Dfalse; offset=3D0;
> locs=3D[DatanodeInfoWithStorage[10.20.84.30:50011
> ,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK],
> DatanodeInfoWithStorage[10.20.84.31:50011
> ,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK],
> DatanodeInfoWithStorage[10.20.84.27:50011
> ,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]]}]
>
>
> lastLocatedBlock=3DLocatedBlock{BP-410607956-10.20.84.26-1391491814882:bl=
k_1078238905_1099516142201;
> getBlockSize()=3D108633903; corrupt=3Dfalse; offset=3D0;
> locs=3D[DatanodeInfoWithStorage[10.20.84.27:50011
> ,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK],
> DatanodeInfoWithStorage[10.20.84.31:50011
> ,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK],
> DatanodeInfoWithStorage[10.20.84.30:50011
> ,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]]}
>   isLastBlockComplete=3Dtrue}
> 2015-03-17 21:42:58,376 DEBUG [main] hdfs.DFSClient: Connecting to datano=
de
> 10.20.84.30:50011
> 2015-03-17 21:42:58,381 DEBUG [main] hdfs.DFSClient: Connecting to datano=
de
> 10.20.84.27:50011
>
> Do you see it reading from 'good' or 'bad' blocks?
>
> I added this line to hbase log4j.properties to enable DFSClient DEBUG:
>
> log4j.logger.org.apache.hadoop.hdfs.DFSClient=3DDEBUG
>
> On HBASE-12949, what exception is coming up?  Dump it in here.
>
>
>
> > My goal is to determine whether the block in question is actually corru=
pt
> > and, if so, in what way.
>
>
> What happens if you just try to copy the file local or elsewhere in the
> filesystem using dfs shell. Do you get a pure dfs exception unhampered by
> hbaseyness?
>
>
>
> > If it's possible to recover all of the file except
> > a portion of the affected block, that would be OK too.
>
>
> I actually do not see a 'fix' or 'recover' on the hfile tool. We need to
> add it so you can recover all but the bad block (we should figure how to
> skip the bad section also).
>
>
>
> > I just don't want to
> > be in the position of having to lose all 3 gigs of data in this
> particular
> > region, given that most of it appears to be intact. I just can't find t=
he
> > right low-level tools to let me determine the diagnose the exact state
> and
> > structure of the block data I have for this file.
> >
> >
> Nod.
>
>
>
> > Any help or direction that someone could provide would be much
> appreciated.
> > For reference, I'll repeat that our client is running Hadoop
> 2.0.0-cdh4.6.0
> > and add that the HBase version is 0.94.15-cdh4.6.0.
> >
> >
> See if any of the above helps. I'll try and dig up some more tools in
> meantime.
> St.Ack
>
>
>
> > Thanks!
> >
> > -md
> >
>


--=20
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

--001a11359ede9ed427051194701c--