apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandesh Hegde <sand...@datatorrent.com>
Subject Re: Bleeding edge branch ?
Date Wed, 20 Jul 2016 17:43:35 GMT
@Amol

EOL is important for master branch. To start the work on next version of
Hadoop on different branch ( let us call that master++ ), we should not
worry about the EOL. Eventually, master++ becomes master and the master++
will continue on the later version of the Hadoop.



On Wed, Jul 20, 2016 at 10:30 AM Siyuan Hua <siyuan@datatorrent.com> wrote:

> Ok, whether branches or forks. I still think we should have at least some
> materialized version of malhar/core for the big influencer like java,
> hadoop or even kafka. Java 8, for example, is actually not new.  We don't
> have to be aggressive to try out new features from those right now. But we
> can at least have some CI run build/test periodically and make sure our
> current code is future-prove and avoid some future-deprecated code when we
> add new features. Also if people ask for it, we can have a link to point
> them to.  BTW, High-level API can definitely benefit from java 8.  :)
>
> Regards,
> Siyuan
>
> On Wed, Jul 20, 2016 at 8:30 AM, Sandesh Hegde <sandesh@datatorrent.com>
> wrote:
>
> > Our current model of supporting the oldest supported Hadoop, penalizes
> the
> > users of latest Hadoop versions by favoring the slow movers.
> > Also, we won't benefit from the increased maturity of the Hadoop
> platform,
> > as we will be working on the many years old version of Hadoop.
> > We also need to incentivize our customers to upgrade their Hadoop
> version,
> > by making use of new features.
> >
> > My vote goes to start the work on the Hadoop 2.6 ( or any other version )
> > in a different branch, without waiting for the EOL policies.
> >
> > On Tue, Jul 12, 2016 at 1:16 AM Thomas Weise <thomas@datatorrent.com>
> > wrote:
> >
> > > -0
> > >
> > > I read the thread twice, it is not clear to me what benefit Apex users
> > > derive from this exercise. A branch normally contains development work
> > that
> > > is eventually brought back to the main line and into a release. Here,
> the
> > > suggestion seems to be an open ended effort to play with latest tech,
> > isn't
> > > that something anyone (including a group of folks) can do in a fork. I
> > > don't see value in a permanent branch for that, who is going to
> maintain
> > > such code and who will ever use it?
> > >
> > > There was a point that we can find out about potential problems with
> > later
> > > versions. The way to find such issues is to take the releases and run
> > them
> > > on these later versions (that's what users do), not by changing the
> code!
> > >
> > > Regarding Java version: Our users don't use Apex in a vacuum. Please
> > have a
> > > look at ASF Hadoop and the distros EOL policies. That will answer the
> > > question what Java version is appropriate. I would be surprised if
> > > something that works on Java 7 falls flat on the face with Java 8 as a
> > lot
> > > of diligence goes into backward compatibility. Again the way to tests
> > this
> > > is to run verification with existing Apex releases on Java 8 based
> stack.
> > >
> > > Regarding Hadoop version: This has been discussed off record several
> > times
> > > and there are actual JIRA tickets marked accordingly so that the work
> is
> > > done when we move. It is a separate discussion, no need to mix Java
> > > versions and branching with it. I agree with what David said, if
> someone
> > > can show that we can move up to 2.6 based on EOL policies and what
> known
> > > Apex users have in production, then we should work on that upgrade. The
> > way
> > > I imagine it would work is that we have a Hadoop-2.6 (or whatever
> > version)
> > > branch, make all the upgrade related changes there (which should be a
> > list
> > > of JIRAs) and then merge it back to master when we are satisfied. After
> > > that, the branch can be deleted.
> > >
> > > Thomas
> > >
> > >
> > >
> > > On Tue, Jul 12, 2016 at 8:36 AM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > I'm -0 on this idea.
> > > >
> > > > Here is the reason:
> > > > Unless we see a real case where users want to see everything on
> latest,
> > > > this branch might quickly become low hanging fruit and eventually get
> > > > obsolete because its anyway a "no gaurantee" branch.
> > > >
> > > > We have a bunch of dependencies which we'll have to take care of to
> > > really
> > > > make it bleeding edge. Specially about malhar, its a long list. That
> > > looks
> > > > like quite significant work.
> > > > Moreover, if this branch is going to be in "may or may not work"
> state;
> > > I,
> > > > as a user or developer, would bank on what certainly works.
> > > >
> > > > I also think that, if its going to be "no gaurantee" then its worth
> > > > spending time contributions towards master rather than bleeding-edge
> > > > branch.
> > > >
> > > > If a question of "should we upgrade?" comes, the community is mature
> to
> > > > take that call then and work accordingly.
> > > >
> > > > -Chinmay.
> > > >
> > > >
> > > >
> > > > On Tue, Jul 12, 2016 at 11:42 AM, Priyanka Gugale <priyag@apache.org
> >
> > > > wrote:
> > > >
> > > > > +1 for creating such branch.
> > > > > One of us will have to rebase it with master branch at intervals.
I
> > > don't
> > > > > think everyone will cherry-pick their commits here. We can make it
> > once
> > > > in
> > > > > a month activity. Are we considering updating all dependency
> library
> > > > > version as well?
> > > > >
> > > > > -Priyanka
> > > > >
> > > > > On Tue, Jul 12, 2016 at 2:34 AM, Munagala Ramanath <
> > > ram@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Following up on some comments, wanted to clarify what I have
in
> > mind
> > > > for
> > > > > > this branch:
> > > > > >
> > > > > > 1. The main goal is to stay up-to-date with new releases, so
if a
> > > > > question
> > > > > > of the form
> > > > > >     "A new release of X is available, should we upgrade ?" comes
> > up,
> > > > the
> > > > > > answer is
> > > > > >     *always* an *emphatic* yes; otherwise it doesn't bleed enough
> > > (:-)
> > > > as
> > > > > > Sanjay points out.
> > > > > > 2. Pull requests are submitted as always; there is no requirement
> > to
> > > > > > generate an additional
> > > > > >     pull requests against this branch. It may get
> > > merged/cherry-picked
> > > > > > depending on who has the
> > > > > >    time and inclination to do it.
> > > > > > 3. There is no expectation of dedication of any additional
> > resources,
> > > > so
> > > > > > people work on
> > > > > >     it as and when time is available. ("No guarantee" means
> exactly
> > > > > that).
> > > > > > So there is no
> > > > > >     question of "maintaining" this branch.
> > > > > > 4. This branch is not to be encumbered with legacy and/or
> backward
> > > > > > compatibility issues.
> > > > > > 5. This branch is not an experimental sandbox to try out new
> > > > algorithms,
> > > > > > architectural changes
> > > > > >     and other such changes.
> > > > > >
> > > > > > As always, I'm open to other ideas, but that is what I had in
> mind
> > > > when I
> > > > > > made the suggestion.
> > > > > >
> > > > > > Ram
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 11, 2016 at 1:45 PM, Sanjay Pujare <
> > > sanjay@datatorrent.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > As the name suggests the "bleeding-edge" branch ideally
should
> > use
> > > > > > bleeding
> > > > > > > edge versions so I would like to see Java 8 used there
(and
> > Hadoop
> > > 3
> > > > > when
> > > > > > > it does eventually come out) to make the maintenance effort
> > > > > worthwhile...
> > > > > > >
> > > > > > > On Mon, Jul 11, 2016 at 12:05 PM, David Yan <
> > david@datatorrent.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > I'm -0 on Java 8, but I'm +1 on the rest, and I'm
especially
> > > strong
> > > > > +1
> > > > > > > for
> > > > > > > > upgrading the Hadoop dependency version.
> > > > > > > >
> > > > > > > > Here are my reasons:
> > > > > > > >
> > > > > > > > - Hadoop 3 will require Java 8, but Hadoop 2.7.2 still
> supports
> > > > Java
> > > > > 7
> > > > > > > and
> > > > > > > > there will probably be some time (I'm guessing more
than one
> > > year)
> > > > > for
> > > > > > > > Hadoop 3 to become GA and for major distros to support
Hadoop
> > 3.
> > > > The
> > > > > > > > maintenance effort for having two branches, one for
Java 7
> and
> > > one
> > > > > for
> > > > > > > Java
> > > > > > > > 8 is not worth it at this time.
> > > > > > > >
> > > > > > > > - Apex currently uses Hadoop 2.2 dependencies, marked
> > "provided".
> > > > And
> > > > > > > > Hadoop 2.4 has been released more than two years ago,
and it
> > > added
> > > > a
> > > > > > lot
> > > > > > > of
> > > > > > > > features in the API that Apex can make use of. Most
distros
> > > already
> > > > > > > bundle
> > > > > > > > Hadoop 2.6 or later. Although some old versions of
Cloudera
> > that
> > > > > > include
> > > > > > > > hadoop version earlier than 2.4 still have not reached
> > > end-of-life
> > > > > yet,
> > > > > > > the
> > > > > > > > number of users using those old versions is probably
very
> > small.
> > > > > > > >
> > > > > > > > David
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 11, 2016 at 8:59 AM, Munagala Ramanath
<
> > > > > > ram@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We've had a number of issues recently related
to
> dependencies
> > > on
> > > > > old
> > > > > > > > > versions
> > > > > > > > > of various packages/libraries such as Hadoop
itself, Google
> > > > guava,
> > > > > > > > > HTTPClient,
> > > > > > > > > mbassador, etc.
> > > > > > > > >
> > > > > > > > > How about we create a "bleeding-edge" branch
in both Core
> and
> > > > > Malhar
> > > > > > > > which
> > > > > > > > > will use the latest versions of these various
dependencies,
> > > > upgrade
> > > > > > to
> > > > > > > > Java
> > > > > > > > > 8 so
> > > > > > > > > we can use the new Java features, etc. ?
> > > > > > > > >
> > > > > > > > > This will give us an opportunity to discover
these sorts of
> > > > > problems
> > > > > > > > early
> > > > > > > > > and,
> > > > > > > > > when we are ready to pull the trigger for a major
version,
> we
> > > > have
> > > > > a
> > > > > > > > branch
> > > > > > > > > ready
> > > > > > > > > for merge with, hopefully, minimal additional
effort.
> > > > > > > > >
> > > > > > > > > There will be no guarantees w.r.t. this branch
so people
> > using
> > > it
> > > > > use
> > > > > > > it
> > > > > > > > at
> > > > > > > > > their own
> > > > > > > > > risk.
> > > > > > > > >
> > > > > > > > > Ram
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message