hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: [DISCUSS] HBase 3 release plans
Date Mon, 17 Feb 2020 20:37:11 GMT
The flip side, which favors the idea of upgrading, is if all our dependencies or related projects
are moving up but we are not then we are the rock that may prevent a user from upgrading the
whole stack otherwise. 

I was trying to articulate a compromise position earlier: Building for release on 8 retains
bytecode and runtime compatibility with older runtimes, but if we also regularly run integration
tests (and others?) on 11, then we can claim ourselves 11 ready. Perhaps unit testing with
11 alongside 8 for precommit wouldn’t be too onerous. 

> On Feb 17, 2020, at 11:37 AM, Lars Francke <lars.francke@gmail.com> wrote:
> On Mon, Feb 17, 2020 at 8:18 PM Andrew Purtell <andrew.purtell@gmail.com>
> wrote:
>> I realize re reading this I left my point incomplete. Pardon, here is the
>> rest of it:
>> That said, if the consensus emerges that moving up to and taking a
>> dependency on Java 11 is the best thing to do (for benefits ??, at least
>> one being a certification of sorts we run on a LTS > 8), then we
>> acknowledge a lift up to all necessary dependency versions also supporting
>> 11 is part of the scope of work, including some kind of evidence to
>> potential adopters the end result is stable and performs well. I’m not
>> opposed to this project but want to surface what might be some hidden work.
> The dependency issue alone you mentioned is probably a good enough reason
> to not upgrade yet.
> I just outlined some of my and our clients' reasons an upgrade would be
> welcome but it's not _required_  and I won't say more about it.
> If we decide not to do it for 3.0 then we should consider doing it for the
> next version. That's fine with me as well especially if we move to a
> shorter release cycle.
>>> On Feb 17, 2020, at 10:49 AM, Andrew Purtell <andrew.purtell@gmail.com>
>> wrote:
>>> A user can upgrade their runtime to 11 and deploy software built on 8
>> at their option at any time. Staying on 8 doesn’t affect this one way or
>> another.
>>> Are we in the business of Java language evangelism or adoption? No. So
>> as PMC I am -1 on taking on such orthogonal missions.
>>> When you run ZK 3.4’s test suite on 11 about 40% fail. An example.
>> Hadoop 2 versions also have test issues. This doesn’t mean they won’t
>> operate acceptably at runtime on 11 as part of a holistic HBase
>> installation, but it does raise red flags. If we adopt Java 11 all of our
>> transitive dependencies should be testing out green too. That means we
>> upgrade those dependencies or fix them. By “we” I mean the operators who
>> have real world deployments if nobody else if HBase devs shirk the
>> responsibility. I would define our mission, or at least the essential
>> mission of users and operators, as conservative, stable data storage. Red
>> builds of dependencies on 11 does not provide a sense of stability. A bunch
>> of dependency upgrades to compensate does not provide a sense of stability.
>>> The mandating of Java 11 or later runtimes is a much bigger investment
>> that this discussion seems to contemplate. Unless we don’t care about
>> adoption.
>>> If you want next generation low pause GC, Shenandoah has been ported
>> back to 8, is supported and available in RedHat’s OpenJDK, and that JDK
>> ships by default in several modern Linuxen not least of which Amazon Linux
>> AMIs.
>>>> On Feb 17, 2020, at 10:32 AM, Lars Francke <lars.francke@gmail.com>
>> wrote:
>>>> I don't have a super-strong opinion on this but one of the reasons I'm
>> in
>>>> favor of moving to 11 is actually to apply some pressure.
>>>> OpenJDK is in all the major package repositories as far as I know. This
>> is
>>>> different from the Oracle JDK. So it's relatively simple to upgrade.
>>>> Lots of our customers would love to upgrade to newer versions because
>> most
>>>> of the time it means they can use the newer versions also for their
>> Spark
>>>> jobs etc. and this way they have a reason to do so and also some
>> leverage
>>>> in internal politics. I know it's not a technical reason but for a whole
>>>> bunch of people, it's still a very valid one.
>>>> But there are also technical reasons as Sean mentioned:
>>>> * One of them being the whole module system which would be nice to try
>> out
>>>> on HBase (which has a "shady" past in this regard).
>>>> * We could also check whether the Shell could be rewritten using JShell
>>>> (I've never looked into the feasibility but I'd personally love to get
>> away
>>>> from JRuby),
>>>> * Then there's TLS 1.3 support (which honestly might also be backported
>> in
>>>> an earlier version) which is getting more important
>>>> * and we have operational improvements around Garbage Collection which
>>>> would benefit HBase.
>>>>> On Mon, Feb 17, 2020 at 6:57 PM Andrew Purtell <
>> andrew.purtell@gmail.com>
>>>>> wrote:
>>>>> I share this concern.
>>>>> I recommend we build on 8, so can run on any 8 or later runtime, and
>> test
>>>>> on 11 or whatever the desired next advertised version of the runtime
>> will
>>>>> be. Builds and unit tests would execute on 8, then whole stack
>> integration
>>>>> tests on 11 (like ITBLL).
>>>>> Only if a post 8 language feature becomes compelling should there be
>> any
>>>>> need to move up the bytecode compatibility line. Any thoughts there.
>>>>> can’t think of a one. Eventually the value types work might be worth
>> it.
>>>>> There may be differences in runtime, like we’ve seen historically.
>> far
>>>>> we have managed to navigate them just fine by being careful or porting
>> in
>>>>> appropriately licensed support to our own util packages. I would expect
>>>>> this to continue to be possible.
>>>>>> On Feb 17, 2020, at 6:41 AM, 张铎(Duo Zhang) <palomino219@gmail.com>
>>>>> wrote:
>>>>>> For me the only concern is the JDK11+.
>>>>>> As long as lots of big company are starting to make their own OpenJDK8
>>>>>> releases(like AdoptOpenJDK from redhat, and corretto from Amazon,
>> so
>>>>>> on), at least a big part of java world will still be on JDK8, this
>> means
>>>>> we
>>>>>> still need to run HBase 2 for a very long time?
>>>>>> Lars Francke <lars.francke@gmail.com> 于2020年2月17日周一
>>>>>>> Sean,
>>>>>>> you didn't have any explicit questions or request for feedback
>> your
>>>>> mail
>>>>>>> so I'll just say that from my point all the points from your
>> make
>>>>>>> sense to me and I'm +1 on all of them.
>>>>>>> - Timeline (GA December/January)
>>>>>>> - Start of shorter release cycle
>>>>>>> - Rolling upgrade
>>>>>>> - JDK11+
>>>>>>> - Hadoop 3 only
>>>>>>> - No more Log4j 1
>>>>>>> - Minimize Hadoop needs
>>>>>>> Thank you for starting this process!
>>>>>>>>> On Sat, Feb 15, 2020 at 2:55 AM Sean Busbey <busbey@apache.org>
>> wrote:
>>>>>>>>> Hi folks!
>>>>>>>>> Consider this the start of my release manager duties
for HBase
>> 3.0.0.
>>>>>>>>> HBase 2.0 started alpha releases in Jun 2017 and went
GA in April
>>>>>>>>> 2018. I'd like to start alpha releases for HBase 3.0
next week and
>> aim
>>>>>>>>> for a GA in December or January.
>>>>>>>>> As RM, I consider the alpha releases a chance for folks
to shake
>>>>>>>>> things out and for us to decide as a community what makes
it in or
>> not
>>>>>>>>> for the release line. Once things start being labeled
beta, I would
>>>>>>>>> like things to be feature frozen. My current goal is
to set beta in
>>>>>>>>> May or June.
>>>>>>>>> I would like HBase 3.0 to be the start of us getting
into practice
>> on
>>>>>>>>> tighter iteration cycles for major versions. 2.5 years
is too
>> long. We
>>>>>>>>> should try to look at our version numbers as akin to
Linux kernel
>>>>>>>>> version numbers. Something useful to those interested
in the
>> internals
>>>>>>>>> of the project but not something where most downstream
users have
>> to
>>>>>>>>> dread major bumps. To that end I would like HBase 3.0
to be rolling
>>>>>>>>> upgradable from HBase 2.y at GA. Ultimately, I would
like to update
>>>>>>>>> our reference guide's section on compatibility to state
that future
>>>>>>>>> major releases will similarly be rolling upgradable from
some minor
>>>>>>>>> release of the prior major release line.
>>>>>>>>> Given my desire to make major upgrades less of a world
>> event
>>>>>>>>> for our downstream folks, I also don't have any new features
that I
>>>>>>>>> feel strongly need to make it into HBase 3.0. I'll do
a review of
>>>>>>>>> what's currently there so we can motivate folks, but
I won't do
>> that
>>>>>>>>> until we're ready to declare beta since that will be
when I'll
>> have a
>>>>>>>>> better idea of what's actually ready to ship.
>>>>>>>>> With the major version change I'd like us to shed some
>>>>>>>>> burden in the project. For each of these, doing that
will require
>>>>>>>>> getting a branch-2 release out that can successfully
opt-in to the
>>>>>>>>> HBase 3 requirement and run on the current HBase 2 requirement.
>> That
>>>>>>>>> way folks can do a rolling restart within HBase 2 to
make the
>> change,
>>>>>>>>> then do a rolling upgrade to HBase 3.
>>>>>>>>> = Hadoop 3 only
>>>>>>>>> The Hadoop community's focus increasingly is on Hadoop
>> releases. A
>>>>>>>>> substantial amount of our dependency handling is tied
to trying to
>>>>>>>>> span both Hadoop 2 and Hadoop 3. I would like us to drop
Hadoop 2
>>>>>>>>> entirely. I think branch-2 is currently in a place where
running on
>>>>>>>>> Hadoop 3 is reasonable.
>>>>>>>>> = JDK11+
>>>>>>>>> We've been bending over backwards on jdk versions for
a while.
>> Maven
>>>>>>>>> build sets up for multiple jdk builds are a PITA and
our existing
>>>>>>>>> build is already too complicated. I'd like us to get
branch-2 into
>> a
>>>>>>>>> solid state for running on either JDK8 or JDK11 so folks
can do
>>>>>>>>> production upgrades on those releases. I'd like HBase
3 to be
>> jdk11+
>>>>>>>>> only so that we can reduce our test footprint in the
project and
>> start
>>>>>>>>> to entertain features that don't work with jdk8. For
example, we
>> can't
>>>>>>>>> start to figure out how we should fit in the module system
>> things
>>>>>>>>> are.
>>>>>>>>> = No more Log4j 1
>>>>>>>>> We got through 95% of the work to make our logging system
>> pluggable,
>>>>>>>>> but we still ship with log4j 1 as the out of the box
solution. I
>> want
>>>>>>>>> HBase 3 to ship with some other slf4j backend and I want
that same
>>>>>>>>> backend to be a realistic option for branch-2 users to
deploy in
>>>>>>>>> production.
>>>>>>>>> = Minimize Hadoop needs
>>>>>>>>> I would like to isolate the things we have that reach
into Hadoop
>>>>>>>>> internals so that we can ease the logistics of changing
the Hadoop
>>>>>>>>> version we run on and minimize the extra stuff we carry
around for
>>>>>>>>> those who don't run on Hadoop at all. This will involve
moving the
>>>>>>>>> stuff we have that reaches into internals into one or
>> module(s)
>>>>>>>>> and updating our artifacts so that we can tell where
our hadoop
>>>>>>>>> related dependencies are in an installation.

View raw message