hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Date Thu, 06 Oct 2016 19:23:46 GMT
I assume we will have matrix testing of HBase versions against designated
upstream build targets for Hadoop 2.x and Hadoop 3.x? If not, if not too
much trouble would be a good idea. We're ad hoc about checking if a commit
to 0.98 breaks the Hadoop 1 build. Every few release candidates I have to
commit addendums to fix as part of the RC process. For what it's worth.


On Thu, Oct 6, 2016 at 11:58 AM, Jonathan Hsieh <jon@cloudera.com> wrote:

> Again my goal is to have the hadoop 3 profile that already exist provide
> some signal and not be broken as it is currently.  There are are other
> options but this one seems reasonable since hadoop3 releases are starting
> to show up.
>
> The milestone I propose is it compiles properly, and it passes our
> licensing enforcers.  A further milestone for another release is passing
> unit tests.
>
> This is essentially no different for what we do for all the different
> hadoop version (2.6.x 2.7.x etc we support hbase 1.x on.), or the different
> builds with the different jdks.
>
> Jon.
>
> On Thu, Oct 6, 2016 at 10:47 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > bq. one set of hbase binaries that will work against multiple hadoops
> >
> > I would be interested to know what tests are / will be performed
> > against 3.0.0-alpha1
> > (using artifacts built against 2.7.1).
> >
> > Thanks
> >
> >
> > On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh <jon@cloudera.com>
> wrote:
> >
> > > Yes.
> > >
> > > The goal is to produce one set of hbase binaries that will work against
> > > multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but
> > > preferentially tested against and likely including binaries from a
> stable
> > > hadoop version.
> > >
> > > Up until recently, compiling against the hadoop 3 profile fails because
> > of
> > > compilation issues and licensing issues,   Another issue, HBASE-16711
> has
> > > already landed which fixed compilation against hadoop2 and hadoop3.
> What
> > > remains is on the short proposed list makes sure licensing enforcers
> are
> > > satisfied correctly and getting build infrastructure precommit checks
> in
> > > place so we don't inadvertently introduce new problems.
> > >
> > > Jon.
> > >
> > > On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Jon:
> > > > Once the goals you outlined below are achieved, would user be able to
> > use
> > > > build artifacts compiled against hadoop 2.7.1 on a cluster deployed
> > with
> > > > hadoop
> > > > 3.0.0-alpha1 ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh <jon@cloudera.com>
> > > wrote:
> > > >
> > > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and
> > > > passing
> > > > > licensing working for the first hbase alpha (or whatever we end up
> > > > calling
> > > > > it)
> > > > >
> > > > > I'll propose these items:
> > > > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is
> the
> > > > most
> > > > > recent). currently we are against 3.0.0-SNAPSHOT.
> > > > > 2) add precompile checks against a hadoop 3.x (HBASE-16733)
> > > > > 3) get 'mvn test install -Dskiptests' to succeed without licensing
> > > issues
> > > > > (HBASE-16712)
> > > > > 4) Have a job setup in jenkins so that we can gain insight and burn
> > > down
> > > > > unit tests failures against hadoop3.
> > > > >
> > > > > These items have a good chance of landing in the next week or two.
> > > > >
> > > > > Other related issues that are nice to have but wouldn't block an
> > hbase
> > > > > alpha include:
> > > > > 1) having no always failing unit tests against hadoop3 (HBASE-6581)
> > > > >
> > > > > Thoughts?
> > > > > Jon.
> > > > >
> > > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <
> > syuanjiangdev@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hello, All,
> > > > > >
> > > > > > It is time to discuss about the schedule of HBase 2.0 release.
> > HBase
> > > > 2.0
> > > > > > release is a big major release.  When we release 1.0, we had
0.99
> > as
> > > > dev
> > > > > > preview/beta release.  We should do something similar for the
2.0
> > > > > release.
> > > > > >
> > > > > > Matteo and I talked about this.   We think about that we need
> some
> > > > > > Alpha/Beta milestones before the RC and final Release-to-Web
2.0
> > > > release.
> > > > > >
> > > > > > I don't know whether there is any discussion on this community
> > about
> > > > the
> > > > > > Alpha/Beta release criteria.  My proposal is that once we cut
the
> > > > > branch-2,
> > > > > > we should only have new features that are absolutely needed
for
> > major
> > > > > > release (festures cannot be added in minor release) and those
> > > features
> > > > > > should be "almost ready".  For Alpha releases, we can still
> accept
> > > > these
> > > > > > kind of features; for Beta release, only bug fixes and
> performance
> > > > > > improvement on existing features (should we also accept existing
> > > > feature
> > > > > > improvement in Beta?  Maybe Beta 1, Not in Beta 2 - that is
my
> > take).
> > > > > >
> > > > > > This is a big release and requires a lot of work from Release
> > > > Manager.  I
> > > > > > asked Matteo whether I could help to be some kind of backup
/
> > > > > hot-standby /
> > > > > > assistant RM.  I think he is very happy to have someone to share
> > some
> > > > RM
> > > > > > duties.  Thus, I will help make this 2.0 release as smooth as
> > > possible.
> > > > > >
> > > > > > Here is a tentative plan:
> > > > > > - For now, we are thinking of creating branch-2 middle of this
> > month
> > > > and
> > > > > > have 2.0-Alpha1 release at the end of this month or begin of
Nov.
> > > The
> > > > > > definition of Alpha1 is that we could deploy to a cluster and
it
> > can
> > > do
> > > > > > some simple CRUD and table DDLs; and not crash (of course, UT
> > > passing).
> > > > > >
> > > > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1.  It
> would
> > > hold
> > > > > > higher bar.  We will run some IT tests to make sure that it
would
> > > > > > functional.
> > > > > >
> > > > > > - At that time, we will lock down and not allow any new features,
> > > every
> > > > > 4-6
> > > > > > weeks, we will have a Beta release (my realistic guess is that
we
> > > would
> > > > > hit
> > > > > > the US Christmas holiday at that time, so first Beta would take
> > > longer
> > > > > than
> > > > > > 6 weeks).  For Beta release, we would fix bugs and do performance
> > > > tuning.
> > > > > > Planning to have 2 Betas.  (Question: in the past, do we need
> vote
> > to
> > > > > have
> > > > > > a Beta release?)
> > > > > >
> > > > > > - Once the code are in the stable stage, then we will have RCs
> and
> > > vote
> > > > > for
> > > > > > the final release.
> > > > > >
> > > > > > Please let us know whether this is a reasonable approach that
> will
> > > make
> > > > > the
> > > > > > release successful.
> > > > > >
> > > > > > Currently, we are aware of the following on-going new features
> for
> > > 2.0:
> > > > > new
> > > > > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid
> > > > Logical
> > > > > > Clock, and maybe AsyncRegion / C++ client).  If you have a
> feature
> > > that
> > > > > > wants to be part of 2.0 release, please send discussion to dev
> > > > community
> > > > > > and we can make a call on accepting/rejecting.
> > > > > >
> > > > > > Thanks
> > > > > > Stephen
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > // jon@cloudera.com // @jmhsieh
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message