Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of prattrs@adobe.com designates
 64.18.1.37 as permitted sender)
From: Sandy Pratt <prattrs@adobe.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Date: Tue, 31 May 2011 14:27:57 -0700
Subject: RE: HFile.Reader scans return latest version?
Thread-Topic: HFile.Reader scans return latest version?
Thread-Index: AcwfztazaFORFtOXQWeDXPIcHE/tQwAB5N+Q
Message-ID: 
 <A0B699C30D50C84883E79752AEA74B881DB444A7BC@NAMBX01.corp.adobe.com>
References: 
 <A0B699C30D50C84883E79752AEA74B881DB444A6B1@NAMBX01.corp.adobe.com>
 <BANLkTinhrZiasJHXQGbcn-M2Ct=yDCq9KQ@mail.gmail.com>
In-Reply-To: <BANLkTinhrZiasJHXQGbcn-M2Ct=yDCq9KQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks for the pointers.

The damage manifested as scanners skipping over a range in our time series =
data.  We knew from other systems that there should be some records in that=
 region that weren't returned.  When we looked closely we saw an extremely =
improbable jump in rowkeys that should by evenly distributed UUIDs beneath =
an hourly prefix.  We checked the region listing and start/end keys in the =
regionserver UI, and found a region listed that wasn't being served.  We tr=
aced it back to a couple of possible locations under /hbase, and got some o=
dd results when we tried to point the HFile main method at those files.

Here's the region we found missing along with the next one and the previous=
 one:

Previous:
ets.derived.events.pb,2010-09-28-02:dcba1a8d00d945e6a90442c9561e8ac4,128566=
7269423 	ets-lax-prod-hadoop-10.corp.adobe.com:60030 	2010-09-28-02:dcba1a8=
d00d945e6a90442c9561e8ac4 	2010-09-28-05:5457075d4f9345908bdfd89b5b641d3d

Affected region:
ets.derived.events.pb,2010-09-28-05:5457075d4f9345908bdfd89b5b641d3d,128568=
4268773 	ets-lax-prod-hadoop-04.corp.adobe.com:60030 =09
2010-09-28-05:5457075d4f9345908bdfd89b5b641d3d 	2010-09-28-11:29664000a2264=
86e9ecb7547a738d101

Next:
ets.derived.events.pb,2010-09-28-11:29664000a226486e9ecb7547a738d101,128568=
7842817 	ets-lax-prod-hadoop-07.corp.adobe.com:60030 =09
2010-09-28-11:29664000a226486e9ecb7547a738d101 	2010-09-28-12:f8fa9dc21bfe4=
091a4864d0adc655b4d


The affected region on RS UI:
ets.derived.events.pb,2010-09-28-05:5457075d4f9345908bdfd89b5b641d3d,128568=
4268773.1836172434
2010-09-28-05:5457075d4f9345908bdfd89b5b641d3d	2010-09-28-11:29664000a22648=
6e9ecb7547a738d101 	stores=3D1, storefiles=3D1, storefileSizeMB=3D45, memst=
oreSizeMB=3D0, storefileIndexSizeMB=3D0


Directory for region on hdfs (guessing based on suffix from RS UI):
/hbase/ets.derived.events.pb/1836172434


Here's what happened when we ran HFile main method on those files:

Checked with HFile:

[hadoop@ets-lax-prod-hadoop-01 ~]$ hbase org.apache.hadoop.hbase.io.hfile.H=
File -r 'ets.derived.events.pb,2010-09-28-05:5457075d4f9345908bdfd89b5b641d=
3d,1285684268773.1836172434' -v
cat: /opt/hadoop/hbase/target/cached_classpath.txt: No such file or directo=
ry
region dir -> hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:54310/hbase/ets.=
derived.events.pb/107531684
Number of region files found -> 0

Note that it found a different directory on hdfs than I would have thought.=
  Look at that file with HFile and it doesn't like it:

[hadoop@ets-lax-prod-hadoop-01 ~]$ hbase org.apache.hadoop.hbase.io.hfile.H=
File -f hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:54310/hbase/ets.derive=
d.events.pb/107531684 -v -k
cat: /opt/hadoop/hbase/target/cached_classpath.txt: No such file or directo=
ry
Scanning -> hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:54310/hbase/ets.de=
rived.events.pb/107531684
ERROR, file doesnt exist: hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:5431=
0/hbase/ets.derived.events.pb/107531684

Put in the file I thought it was, and although it's there on HDFS, HFile ca=
n't find it:
[hadoop@ets-lax-prod-hadoop-01 ~]$ hbase org.apache.hadoop.hbase.io.hfile.H=
File -f hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:54310/hbase/ets.derive=
d.events.pb/1836172434 -v -k
cat: /opt/hadoop/hbase/target/cached_classpath.txt: No such file or directo=
ry
Scanning -> hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:54310/hbase/ets.de=
rived.events.pb/1836172434
java.io.FileNotFoundException: File does not exist: /hbase/ets.derived.even=
ts.pb/1836172434
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClie=
nt.java:1586)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient=
.java:1577)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:428)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFil=
eSystem.java:185)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:431)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.<init>(HFile.java:=
742)
        at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:1870)

[hadoop@ets-lax-prod-hadoop-01 ~]$ hadoop dfs -ls  /hbase/ets.derived.event=
s.pb/1836172434
Found 2 items
-rw-r--r--   3 hadoop hadoop        862 2010-09-28 07:31 /hbase/ets.derived=
.events.pb/1836172434/.regioninfo
drwxr-xr-x   - hadoop hadoop          0 2011-05-06 16:36 /hbase/ets.derived=
.events.pb/1836172434/f1


Ran an hbase hbck which came back clean.  Stopped HBase and restarted to fi=
nd that hbck gave errors (not sure why it was ok before and not after - may=
be a split happened in the interim or something - but we are running durabl=
e now so hopefully a change to META would not get lost).  After that I made=
 a backup and tried add_table.rb, which seems to make the problem worse.   =
We eventually concluded that we must have lost a write to META last year wh=
en we were running Hadoop 0.20.1 and HBase 0.20.3 without durability (curre=
ntly running CDH3b3).  This is supported by the fact that other environment=
s running the same code are OK and hadoop fsck / is also healthy.

My solution is to create a broadly similar table and read the HFiles from t=
he old one directly into it.  So this would be an MR with an HFileInputForm=
at I wrote using the HFile API, and a TableOutputFormat into the new table =
(didn't want to put writing directly to HFiles on my plate at this time).  =
Once that's done and verified, I'll drop the older table and move on.

Because of the version of HBase we're running, we don' t have hbck -fix ava=
ilable, and I assume it's been months since the damage happened which might=
 mean we have some regions overlapping.  It might be hard to manually stitc=
h them back together, so this holistic approach seemed like the best bet.

One thing I can put as a win in HBase's column is that the damaged table st=
ill functions fine in the parts that don't have holes, which is the majorit=
y of the table.  So we can keep running for the majority of our dataset (an=
d work) and take the time to fix the damage carefully.

Sandy

> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Tuesday, May 31, 2011 13:10
> To: user@hbase.apache.org
> Subject: Re: HFile.Reader scans return latest version?
>=20
> On Tue, May 31, 2011 at 11:05 AM, Sandy Pratt <prattrs@adobe.com> wrote:
> > Hi all,
> >
> > I'm doing some work to read records directly from the HFiles of a damag=
ed
> table. =A0When I scan through the records in the HFile using
> org.apache.hadoop.hbase.io.hfile.HFileScanner, will I get only the latest
> version of the record as with a default HBase Scan? =A0Or do I need to do=
 some
> work to pull out the latest version from several?
> >
>=20
> It looks like it just returns all entries in the hfile.  See tests -- e.g=
. TestHFile --
> for how to make make an HFile Reader instance and pull the values.  The t=
ail
> of HFile has some examples too?
>=20
> Tell us about the 'damaged table'.
>=20
> St.Ack.