hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Robertson <timrobertson...@gmail.com>
Subject Data loss in MOB snapshot and clone?
Date Wed, 12 Oct 2016 15:02:31 GMT
Hi devs,
[Had a quick chat with Lars G. about this and before opening a Jira I
thought I'd raise it here first]

We have just experienced data loss in HBase 1.0.0-cdh5.4.10.

Before I dig into this further, I'd like to just ask if anyone has seen
this before?

The initial state was a table (tim_test) built with MOB support and a few
10's million rows and 10's billions of cells.

I wanted to rename the table to get this into production and did so as

  snapshot 'tim_test', 'tim_test-snapshot'
  clone_snapshot 'tim_test-snapshot', 'prod_b_map'

At this stage the application all looked good, and so I continued with:

  delete_snapshot 'tim_test-snapshot'
  disable 'tim_test'
  drop ‘tim_test’

Then things went... awry and data just started dropping out in the app.
Before long, all MOB data seemingly is gone.

The references in the new table MOB folder appear to point to the source
table (e.g.

The RS logs full of ERROR like:

2016-10-12 15:19:14,640 ERROR org.apache.hadoop.hbase.regionserver.HStore:
The mob file
could not be found in the locations

What I don't know is:
 1) was this running a background task to copy the MOB data when the
snapshot was cloned and I just deleted the source before the copy was
 - or
 2) when running "snapshot and clone" it just references the source MOB
data until a (?) change?
 3) snapshot and clone just doesn't support MOB?

Can anyone shed some light on this easily before I dig into it please?

While this situation exists (at least in 1.0.0) might it be good to get
info about data loss for MOB tables into the snapshot clone docs?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message