Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of ogdude@googlemail.com
 designates 209.85.220.50 as permitted sender)
MIME-Version: 1.0
Date: Fri, 31 Jan 2014 14:06:22 +0100
Message-ID: 
 <CAE24rAdESxYs9LEZKDVTPeDrpEMBtr==WWS3r07UjfrYgca2ww@mail.gmail.com>
Subject: Snapshot clone error
From: David Koch <ogdude@googlemail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=bcaec5215bcb30194104f143d742

--bcaec5215bcb30194104f143d742
Content-Type: text/plain; charset=ISO-8859-1

Hello,

We export an online snapshot of a table to a different cluster, when
attempting a clone on the destination cluster using:

clone_snapshot 'table_source_snapshot', 'table_dest'

it does not work.

The operation times out after a a while

ERROR: java.io.IOException: Table 'table_dest' not yet enabled, after
1996939ms.

and I see only a fraction of the number of regions in the destination
table. Table is indicated as "enabled" but I cannot perform any scans on it.

The snapshot info returns the following:

Snapshot Info
----------------------------------------
   Name: table_source_snapshot
   Type: FLUSH
  Table: table_source
 Format: 0
Created: 2014-01-30T13:05:02

Snapshot seems to be intact. What could be the error? Should I take an
offline snapshot instead? Going via restore/enable instead of clone does
not seem to work either.

Also, I see the following in the region servers:

2:24:35.807 PM ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler
Failed open of
region= table_source,\x82\x12Y\x00\xE98C\xEE\xBC\xCC\xE3h\xDAPt\xA6,1366259070788.63ca017ac7cd03e68c35a4da8b56421d.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
Unable to open link: org.apache.hadoop.hbase.io.HFileLink
locations=[hdfs://nameservice1/hbase/table_source/816bb88c6f3524a877f4cb7ce747fec1/t/c3b37dc11e684626a5b464a25a75735c,
hdfs://nameservice1/hbase/.tmp/table_source/816bb88c6f3524a877f4cb7ce747fec1/t/c3b37dc11e684626a5b464a25a75735c,
hdfs://nameservice1/hbase/.archive/table_source/816bb88c6f3524a877f4cb7ce747fec1/t/c3b37dc11e684626a5b464a25a75735c]

None of these parts actually exist, however:
hdfs://nameservice1/hbase/.snapshot/table_source_snapshot/816bb88c6f3524a877f4cb7ce747fec1/t/c3b37dc11e684626a5b464a25a75735c
does exist.

I don't think that's the issue though, since I applied the same steps to a
smaller table and it worked.

Any advice is appreciated,

Regards,

/David

--bcaec5215bcb30194104f143d742--