hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matteo Bertozzi <theo.berto...@gmail.com>
Subject Re: Copying data from 94 to 98 ..
Date Tue, 16 Sep 2014 06:12:56 GMT
ok, so you have an empty reference file under .snapshot/SnapshotName/...
what you can do is find all the reference files and replace them with the
version in /hbase/.archive
you can find reference files because are in the form "name.regionName"
(they are the only files with a dot in the middle)

so.. something like
cp /hbase/.archive/tableName/region/myfile.parent
 /hbase/.snapshot/snapshotName/region/myfile.parent



Matteo


On Mon, Sep 15, 2014 at 11:00 PM, Gautam <gautamkowshik@gmail.com> wrote:

> Jerry,
>           Can you elaborate on what you mean by "export table to hdfs"?  I
> initially tried running the export on src cluster (-copy-to
> hdfs://dest/hbase ), it complains while trying to write the data to dest
> cluster (due to the hdfs protocol version mismatch). Then I tried running
> export on dest cluster (-copy-from hftp://src/hbase).
>
> On Mon, Sep 15, 2014 at 10:36 PM, Jerry He <jerryjch@gmail.com> wrote:
>
> > While you continue on the snapshot approach, have you tried to Export the
> > table in 0.94 to hdfs, and then Import the data from hdfs to 0.98?
> > On Sep 15, 2014 10:19 PM, "Matteo Bertozzi" <theo.bertozzi@gmail.com>
> > wrote:
> >
> > > can you post the full exception and the file path ?
> > > maybe there is a bug in looking up the reference file.
> > > It seems to not be able to find enough data in the file...
> > >
> > > Matteo
> > >
> > >
> > > On Mon, Sep 15, 2014 at 10:08 PM, Gautam <gautamkowshik@gmail.com>
> > wrote:
> > >
> > > > Thanks for the reply Matteo.
> > > >
> > > > This is exactly what I did. I modified the source cluster's dir
> > structure
> > > > to mimic that of the 98 cluster. I even got as far as it trying to
> look
> > > > through the reference files.
> > > >
> > > > I end up with this exception :
> > > >
> > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> failed
> > > > java.io.IOException
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > ...
> > > > ..
> > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > ..
> > > >
> > > >
> > > > This and Ted's reply about HBASE-7987 leads me to believe that the
> > export
> > > > tool from my distro is in capable of working around the
> regionManifest
> > > file
> > > > requirement. I'm now left with the option of downgrading my dest
> > cluster
> > > to
> > > > 94, copying data and then upgrading using the upgrade migration tool.
> > > > Wanted to know if others have tried this or there are other things I
> > can
> > > > do. If not, i'l just go ahead and do this :-)
> > > >
> > > >
> > > > Cheers,
> > > > -Gautam.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <
> > > theo.bertozzi@gmail.com>
> > > > wrote:
> > > >
> > > > > 94 and 98 differs in directory layout
> > > > > so 98 is not able to read 94 layout unless you run the migration
> tool
> > > > > which is basically moving all the data in a "default" namespace
> > > directory
> > > > > e.g.
> > > > > /hbase/table -> /hbase/data/default/table
> > > > > /hbase/.archive/table -> /hbase/archive/default/table
> > > > >
> > > > > Matteo
> > > > >
> > > > >
> > > > > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <gautamkowshik@gmail.com>
> > > wrote:
> > > > >
> > > > > > Yep, looks like the CDH distro backports HBASE-7987. Having
said
> > > that,
> > > > is
> > > > > > there a transition path for us or are we hosed :-) ? In general,
> > > what's
> > > > > the
> > > > > > recommended way to achieve this, at this point I feel i'm going
> > > around
> > > > > the
> > > > > > system to achieve what I want. If nothing else works with export
> > > > snapshot
> > > > > > should I just downgrade to 94, export snapshot and then upgrade
> to
> > > 98?
> > > > Is
> > > > > > the upgrade migration path different from what export snapshot
> does
> > > > (i'd
> > > > > > imagine yes)?
> > > > > >
> > > > > > Cheers,
> > > > > > -Gautam.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > > >
> > > > > > > bq. 98.1 on dest cluster
> > > > > > >
> > > > > > > Looking at the history for SnapshotManifestV1, it came
with
> > > > HBASE-7987
> > > > > > > which went to 0.99.0
> > > > > > >
> > > > > > > Perhaps you're using a distro with HBASE-7987 ?
> > > > > > >
> > > > > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <
> gautamkowshik@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >           I'm trying to copy data between Hbase clusters
on
> > > > different
> > > > > > > > versions. I am using :
> > > > > > > >
> > > > > > > > /usr/bin/hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > > > > >   -chuser hbase
> > > > > > > >   -chgroup hadoop
> > > > > > > >   -snapshot msg_snapshot
> > > > > > > >   -mappers 50
> > > > > > > >   -copy-from hftp://src-cluster:50070/hbase
> > > > > > > >   -copy-to hdfs:/dest-cluster:8020/hbase
> > > > > > > >
> > > > > > > >
> > > > > > > > Till now, based on various tips from the mailing list,
I have
> > > > > modified
> > > > > > > the
> > > > > > > > source cluster data dir paths to mimic the 98 convention
> > > (archive,
> > > > > > table
> > > > > > > > data paths, etc). This helped in jumping some roadblocks
but
> > not
> > > > all.
> > > > > > > >
> > > > > > > > This is what I see now :
> > > > > > > >
> > > > > > > >
> > > > > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot
> > export
> > > > > failed
> > > > > > > > java.io.IOException
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > > > > > ...
> > > > > > > > ..
> > > > > > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > > > > > at
> > org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > > > > > ..
> > > > > > > >
> > > > > > > >
> > > > > > > > Fails while trying to read refernce hfile. Is this
something
> > > folks
> > > > > have
> > > > > > > > done before and/or is possible to do? I'd really like
to do
> > this
> > > > > > without
> > > > > > > > having to upgrade my source cluster or downgrade my
dest
> > cluster.
> > > > > > > >
> > > > > > > > I'm using 94.6 on source cluster and 98.1 on dest
cluster.
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > -Gautam.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > "If you really want something in this life, you have to work
for
> > it.
> > > > Now,
> > > > > > quiet! They're about to announce the lottery numbers..."
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > "If you really want something in this life, you have to work for it.
> > Now,
> > > > quiet! They're about to announce the lottery numbers..."
> > > >
> > >
> >
>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message