Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
References: <CADcMMgHpEYHBUVYZfp1VhWob7B=ayC8DwTBE7bje_ZZ9Ff1-nQ@mail.gmail.com>
 <CAN+qs_NsAy-bc13xgdBwRSDoKQ-QyMOvYsrH73gP-_yCkpQeXg@mail.gmail.com>
 <CADcMMgEmkSqWZ-UPbPWBXqesK8CbZn1QwCp-y+EG=NWbSyzMPA@mail.gmail.com>
 <CAN+qs_PkFzp1Niiha_fu50d3h=c_oO9uNLG5g4OcmJPK1KoZ8w@mail.gmail.com>
 <CAN+qs_P5cNAOutc_is66nbnbfsJcJXMaC-89FsKygxVgm8=e2Q@mail.gmail.com>
 <CALte62yDTS6Cwt+AVV_MEn65oq=FyqoMo6z3A_3Cx=bRoRZ5qQ@mail.gmail.com> <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-fc5e8e9b4fccf4a0b027882ae68635ba85f5eae4@dev.hbase.apache.org>
In-Reply-To: <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-fc5e8e9b4fccf4a0b027882ae68635ba85f5eae4@dev.hbase.apache.org>
From: Francis Christopher Liu <toffer.liu@gmail.com>
Date: Tue, 05 Sep 2017 01:55:52 +0000
Message-ID: <CAH4uCE+yBwMDNUYahmBQqihNWUtxqQzN_r2L-KcfT8XftmQuyA@mail.gmail.com>
Subject: Re: [DISCUSS] hbase-2.0.0 compatibility expectations
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary="f403045d25568ce66405586787c2"
archived-at: Tue, 05 Sep 2017 01:56:12 -0000

--f403045d25568ce66405586787c2
Content-Type: text/plain; charset="UTF-8"

Just some high-level thoughts on rolling upgradeability (I'm repeating a
few things already said).

1. No service interruption for 1.x clients while a cluster is being
ugpraded to 2.x.
  - This includes support for apis commonly used in a running system:
delegation tokens, getting region locations, etc.
2. The somewhat reverse of #1 is also necessary for Master DMLs to meta.
3. No interruption for replication between 1.x and 2.x cluster
4. Restart master first before any other service sounds reasonable to me.
Will we support RU for zkless mode?
5. 2.x should be either able to understand formats written by 1.x or online
migration is done as a preparation step.
6. Data generated by 2.x daemons should be readable in 1.x (ie HFile,
zookeeper, etc). Some deploys may have large amounts of data could be
cumbersome/impractical (ie cost, time, etc) to copy the data.

Thanks,
Francis


On Fri, Sep 1, 2017 at 10:01 PM Chia-Ping Tsai <chia7712@apache.org> wrote:

> The most of issues use hadoop flag, so the filter looks like this.
> project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2,
> 2.0.0-alpha-3,
> 3.0) AND (labels in (incompatibleChange, incompatible, incompatibility) OR
> "Hadoop Flags" in ("Incompatible change"))
>
> On 2017-08-04 05:46, Ted Yu <yuzhihong@gmail.com> wrote:
> > I expanded the condition in the filter like this:
> >
> > project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1,  2.0.0-alpha-2,
> > 3.0) AND labels in (incompatibleChange, incompatible, incompatibility)
> >
> > Still there're only two showing up.
> >
> > On Thu, Aug 3, 2017 at 9:07 AM, Zach York <zyork.contribution@gmail.com>
> > wrote:
> >
> > > I tried to filter based on imcompatible labels and there were only two
> > > JIRAs returned [1]. I have a hard time believing that there were only
> two
> > > breaking changes from 1.x to 2.x.
> > >
> > > [1]
> > > https://issues.apache.org/jira/browse/HBASE-17957?jql=
> > > project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%
> > > 20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%
> > > 20incompatibility)
> > >
> > > On Thu, Aug 3, 2017 at 8:54 AM, Zach York <
> zyork.contribution@gmail.com>
> > > wrote:
> > >
> > > > This kinda helps, but these seem more like expectations. I was going
> more
> > > > for things like HFile format changed, meta table structure changed,
> > > > coprocessor implementations changed (these are just examples, I don't
> > > know
> > > > if any of these actually changed).
> > > >
> > > > More technical differences between branch-1 and branch-2 which then
> can
> > > > help us get the right expectations for compatibility.
> > > >
> > > > On Wed, Aug 2, 2017 at 6:34 PM, Stack <stack@duboce.net> wrote:
> > > >
> > > >> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> zyork.contribution@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > Do we know what the major pain points for migration are? Can we
> > > discuss
> > > >> > that/get a list going?
> > > >> >
> > > >> >
> > > >> Here's a few in outline:
> > > >>
> > > >> + There is issue of formats, of hbase-2.x being able to read
> hbase-1.x
> > > >> data
> > > >> whether from HDFS or ZooKeeper or off the wire.
> > > >> + An hbase-1.x client should be able to Get/Put and Scan an
> hbase-2.x
> > > >> cluster; no holes in the API or unintelligible serializations.
> > > >> + There is then the little dance that has us rolling restart from an
> > > >> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then
> it
> > > will
> > > >> assign regions to the new hbase-2.x regionservers as they come on
> line.
> > > >> TBD.
> > > >>
> > > >> Is this what you mean sir?
> > > >>
> > > >> S
> > > >>
> > > >>
> > > >> > I think without that knowledge it is hard (for me at least :) ) to
> > > >> > determine where we should set our sights in terms of migration.
> > > >> >
> > > >> > Thanks,
> > > >> > Zach
> > > >> >
> > > >> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <stack@duboce.net> wrote:
> > > >> >
> > > >> > > What are our expectations regards compatibility between hbase1
> and
> > > >> > hbase2?
> > > >> > >
> > > >> > > Lets have a chat about it. Here are some goal posts.
> > > >> > >
> > > >> > > + You have to upgrade to hbase-1.x before you can migrate to
> > > hbase-2.
> > > >> No
> > > >> > > migration from < hbase-1 (Is this too onerous? Should we support
> > > 0.98
> > > >> =>
> > > >> > > 2.0?).
> > > >> > > + You do NOT have to upgrade to the latest release of hbase1 to
> > > >> migrate
> > > >> > to
> > > >> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > >> > > + You'll have to update your hbase1 coprocessors to deploy them
> on
> > > >> > hbase2.
> > > >> > > A bunch of CP API has/will change by the time hbase2 comes out;
> e.g.
> > > >> > > watching for region split on RegionServer no longer makes sense
> > > given
> > > >> > > Master runs all splits now.
> > > >> > > + An hbase1 client can run against an hbase2 cluster but it will
> > > only
> > > >> be
> > > >> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able
> to
> > > do
> > > >> > admin
> > > >> > > ops using an hbase1 Admin client against an hbase2 cluster. We
> have
> > > >> some
> > > >> > > egregious API violations in branch-1; e.g. we have protobuf in
> our
> > > API
> > > >> > (See
> > > >> > > HBASE-15607). The notion is that we can't afford a deprecation
> cycle
> > > >> > > purging this stuff from our Admin API.
> > > >> > >
> > > >> > > What you all think?
> > > >> > >
> > > >> > > St.Ack
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

--f403045d25568ce66405586787c2--