hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Date Mon, 02 Jan 2017 07:29:56 GMT
Ted,

2016-12-30 17:39 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:

> For hbase094_hbase_default.xml, currently it is
> under hbase094/src/main/resources
> Looks like it belongs to the test directory.
>


Yes because for now I have used a JUnit test in order to run the project.
I will write done some command-line tool in order to run the full migration

If you or other users are interested in such a tool you can write down some
use-case and I will be happy to keep it in consideration.




>
> bq. use the 'Configuration#set" method to programmatically change the
> values
>

Maybe I will let the user pass its own hbase-site.xml file, but instead of
loading directly I will load into memory and then call
HBaseConfiguration.set(key,value)  for each entry.
The way HBaseConfiguration looks ho hbase-site.xml is somehow tricky
because it happens in the constructor and so even with subclassing it is
not possible to disable the 'version check'



> For end user, it would be more convenient to pass hbase-site.xml
>
> Cheers
>
> On Fri, Dec 30, 2016 at 8:19 AM, Enrico Olivelli <eolivelli@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > 2016-12-30 17:11 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:
> >
> > > For scanHBase094Table():
> > >
> > >         baseDefaults.addResource("hbase094_hbase_default.xml");
> > >
> > > where hbase.rootdir points to file://
> > >
> >
> > > How is the user supposed to plug in hbase-site.xml for the 0.94
> cluster ?
> > >
> >
> > the resource gets loaded from the classpath by the Hadoop Configuration
> > API.
> >
> > This is just a code sample to prove that a JVM can connect to two
> > incompatible versions of HBase.
> >
> > you can plug in your configuration by loading a "Configuration" built by
> > reading your configuration file.
> > Personally I do not use the .xml files for configuration of clients but
> use
> > the 'Configuration#set" method to programmatically change the values
> >
> >
> > >
> > > For TableMigrationManager, I don't see where setBatchSize() is called.
> > Does
> > > this mean that batch size of 0 would be in effect ?
> > >
> >
> > It is a bug, so the API uses its default values. See the HBase API docs
> >
> >
> > >
> > > Do you use any API specific to 1.2.4 ? Otherwise I suggest renaming
> > > hbase124
> > > as hbase1 since 1.x releases are compatible.
> > >
> >
> > +1
> > The only API that I use are the same as for 0.94 and 1.2
> > (scan,get,put,delete.....)
> >
> >
> > > Thanks
> > >
> > >
> > >
> > > On Fri, Dec 30, 2016 at 6:33 AM, Enrico Olivelli <eolivelli@gmail.com>
> > > wrote:
> > >
> > > > Thanks you Duo Zhang for the interest.
> > > >
> > > > The example comes from a Java class which actually "migrates" data
> > > > from one table in a 0.94 cluster to a 1.2.4 cluster
> > > >
> > > > this is the file:
> > > > https://github.com/eolivelli/multi-hbase/blob/master/multi-
> > > > hbase-client/src/test/java/MigrateHBaseDataExample.java
> > > >
> > > > this can be the foundation for a future migration tool but at this
> > > > stage that project is only a proof-of-concept
> > > >
> > > >
> > > > 2016-12-30 14:22 GMT+01:00 张铎(Duo Zhang) <palomino219@gmail.com>:
> > > > > Nice try!
> > > > >
> > > > > Is this a migration tool? Or a thin layer on top of the original
> > hbase
> > > > > client to make the user be able to access both 0.94 and 1.2.4 with
> > the
> > > > same
> > > > > API?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eolivelli@gmail.com>:
> > > > >
> > > > >> Hi all,
> > > > >> The migration from 0.94 to 1.2.4 is really a pain and  some time
> > ago I
> > > > >> asked to this list if there was a way to connect to both a 0.94
> > > > >> cluster and a 1.2.4 cluster.
> > > > >>
> > > > >> I have created a proof-of-concept project which is able to achieve
> > my
> > > > >> goal but "shading" most of the 0.94 client.
> > > > >>
> > > > >> This is the source code and the list of tricks to make it work:
> > > > >> https://github.com/eolivelli/multi-hbase
> > > > >>
> > > > >> The above code simply performs a full scan from a 0.94 table
and
> > > > >> performs puts to a 1.2.4 cluster, but I think that most of the
> basic
> > > > >> API of the 0.94 client will work and the full API of the 1.2.4
is
> > > > >> available.
> > > > >>
> > > > >> I'm going to implement a more stable version of the client to
be
> > used
> > > > >> in production.
> > > > >>
> > > > >> I will appreciate a lot If anyone has some time to take a look
and
> > > > >> give me some feedback.
> > > > >>
> > > > >> Cheers
> > > > >> Enrico Olivelli
> > > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message