hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gautam <gautamkows...@gmail.com>
Subject Re: Copying data from 94 to 98 ..
Date Tue, 16 Sep 2014 05:08:55 GMT
Thanks for the reply Matteo.

This is exactly what I did. I modified the source cluster's dir structure
to mimic that of the 98 cluster. I even got as far as it trying to look
through the reference files.

I end up with this exception :

14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
java.io.IOException
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
...
..
Caused by: java.io.IOException: read=-1, wanted=4
at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
..


This and Ted's reply about HBASE-7987 leads me to believe that the export
tool from my distro is in capable of working around the regionManifest file
requirement. I'm now left with the option of downgrading my dest cluster to
94, copying data and then upgrading using the upgrade migration tool.
Wanted to know if others have tried this or there are other things I can
do. If not, i'l just go ahead and do this :-)


Cheers,
-Gautam.



On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
wrote:

> 94 and 98 differs in directory layout
> so 98 is not able to read 94 layout unless you run the migration tool
> which is basically moving all the data in a "default" namespace directory
> e.g.
> /hbase/table -> /hbase/data/default/table
> /hbase/.archive/table -> /hbase/archive/default/table
>
> Matteo
>
>
> On Mon, Sep 15, 2014 at 6:17 PM, Gautam <gautamkowshik@gmail.com> wrote:
>
> > Yep, looks like the CDH distro backports HBASE-7987. Having said that, is
> > there a transition path for us or are we hosed :-) ? In general, what's
> the
> > recommended way to achieve this, at this point I feel i'm going around
> the
> > system to achieve what I want. If nothing else works with export snapshot
> > should I just downgrade to 94, export snapshot and then upgrade to 98? Is
> > the upgrade migration path different from what export snapshot does (i'd
> > imagine yes)?
> >
> > Cheers,
> > -Gautam.
> >
> >
> >
> >
> > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > bq. 98.1 on dest cluster
> > >
> > > Looking at the history for SnapshotManifestV1, it came with HBASE-7987
> > > which went to 0.99.0
> > >
> > > Perhaps you're using a distro with HBASE-7987 ?
> > >
> > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <gautamkowshik@gmail.com>
> wrote:
> > >
> > > > Hello,
> > > >           I'm trying to copy data between Hbase clusters on different
> > > > versions. I am using :
> > > >
> > > > /usr/bin/hbase  org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > >   -chuser hbase
> > > >   -chgroup hadoop
> > > >   -snapshot msg_snapshot
> > > >   -mappers 50
> > > >   -copy-from hftp://src-cluster:50070/hbase
> > > >   -copy-to hdfs:/dest-cluster:8020/hbase
> > > >
> > > >
> > > > Till now, based on various tips from the mailing list, I have
> modified
> > > the
> > > > source cluster data dir paths to mimic the 98 convention (archive,
> > table
> > > > data paths, etc). This helped in jumping some roadblocks but not all.
> > > >
> > > > This is what I see now :
> > > >
> > > >
> > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> failed
> > > > java.io.IOException
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > ...
> > > > ..
> > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > ..
> > > >
> > > >
> > > > Fails while trying to read refernce hfile. Is this something folks
> have
> > > > done before and/or is possible to do? I'd really like to do this
> > without
> > > > having to upgrade my source cluster or downgrade my dest cluster.
> > > >
> > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > >
> > > >
> > > > Cheers,
> > > > -Gautam.
> > > >
> > >
> >
> >
> >
> > --
> > "If you really want something in this life, you have to work for it. Now,
> > quiet! They're about to announce the lottery numbers..."
> >
>



-- 
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message