Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
From: Tim Robertson <timrobertson100@gmail.com>
Date: Wed, 12 Oct 2016 17:02:31 +0200
Message-ID: <CAMsy_Nj6fP1YiRMB=ja7YktoWoFztK1UaAutCbVG6OnbigOhjw@mail.gmail.com>
Subject: Data loss in MOB snapshot and clone?
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11352ee64ea69a053eac486d
archived-at: Wed, 12 Oct 2016 15:02:40 -0000

--001a11352ee64ea69a053eac486d
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi devs,
[Had a quick chat with Lars G. about this and before opening a Jira I
thought I'd raise it here first]

We have just experienced data loss in HBase 1.0.0-cdh5.4.10.

Before I dig into this further, I'd like to just ask if anyone has seen
this before?

The initial state was a table (tim_test) built with MOB support and a few
10's million rows and 10's billions of cells.

I wanted to rename the table to get this into production and did so as
follows:

  snapshot 'tim_test', 'tim_test-snapshot'
  clone_snapshot 'tim_test-snapshot', 'prod_b_map'

At this stage the application all looked good, and so I continued with:

  delete_snapshot 'tim_test-snapshot'
  disable 'tim_test'
  drop =E2=80=98tim_test=E2=80=99

Then things went... awry and data just started dropping out in the app.
Before long, all MOB data seemingly is gone.

The references in the new table MOB folder appear to point to the source
table (e.g.
/hbase/mobdir/data/default/prod_b_map/ba42a2e8e9b669d9fc85bdfeed2f5f2a/EPSG=
_4326/tim_test=3D14bf5f1737ac65c34615ed97c0b7de06-d41d8cd98f00b204e9800998e=
cf8427e20161006ff8baa70d21f408caefe8ae6318dfba2).

The RS logs full of ERROR like:

2016-10-12 15:19:14,640 ERROR org.apache.hadoop.hbase.regionserver.HStore:
The mob file
d41d8cd98f00b204e9800998ecf8427e20161006b59865f80e604781a79ebfa2ddd66b48
could not be found in the locations
[hdfs://ha-nn/hbase/mobdir/data/default/tim_test/14bf5f1737ac65c34615ed97c0=
b7de06/EPSG_4326,
hdfs://ha-nn/hbase/archive/data/default/tim_test/14bf5f1737ac65c34615ed97c0=
b7de06/EPSG_4326]

What I don't know is:
 1) was this running a background task to copy the MOB data when the
snapshot was cloned and I just deleted the source before the copy was
complete?
 - or
 2) when running "snapshot and clone" it just references the source MOB
data until a (?) change?
 3) snapshot and clone just doesn't support MOB?

Can anyone shed some light on this easily before I dig into it please?

While this situation exists (at least in 1.0.0) might it be good to get
info about data loss for MOB tables into the snapshot clone docs?

Thanks,
Tim

--001a11352ee64ea69a053eac486d--