flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: [DISCUSS] Project build time and possible restructuring
Date Wed, 15 Mar 2017 15:44:37 GMT
I'm ok with point 3.

Concerning point 8: Why do we have to build flink-core twice after having
it built as a dependency for flink-libraries? This seems wrong to me.

Cheers,
Till

On Wed, Mar 15, 2017 at 4:23 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Thank you. Running on AWS is a good idea!
> Let me know if you (or anybody else) wants to help me with the
> infrastructure work! Any help is much appreciated (as I've said before, I
> don't really have time for doing this, but it has to be done :) )
>
> I'm against creating two new repositories. I fear that this introduces too
> much complexity and too many repositories.
> "flink" and "flink-libraries" are hopefully enough to get the build time
> significantly down.
> We can also consider putting the connectors into the "flink-libraries" repo
> if we need to further reduce the build time.
>
> We should probably move "flink-table" of out "flink-libraries" if we want
> to keep "flink-table" in the main repo. (This would eliminate the
> "flink-libraries" module from main.
>
> Also, I agree that "flink-statebackend-rocksdb" is not correctly placed in
> contrib anymore.
>
>
> On Wed, Mar 15, 2017 at 4:07 PM, Greg Hogan <code@greghogan.com> wrote:
>
> > Robert, appreciate your kickstarting this task.
> >
> > We should compare the verification time with and without the listed
> > modules. I’ll try to run this by tomorrow on AWS and on Travis.
> >
> > Should we maintain separate repos for flink-contrib and flink-libraries?
> > Are you intending that we move flink-table out of flink-libraries (and
> > perhaps flink-statebackend-rocksdb out of flink-contrib)?
> >
> > Greg
> >
> >
> > > On Mar 15, 2017, at 9:55 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
> > >
> > > Thank you for looking into this Till.
> > >
> > > I think we then have to split the repositories.
> > > My main motivation for doing this is that it seems to be the only
> > feasible
> > > way of scaling the community to allow more committers working on the
> > > libraries.
> > >
> > > I'll take care of getting things started.
> > >
> > > As the next steps I propose to:
> > > 1. Ask INFRA to rename https://git-wip-us.apache.org/
> repos/asf?p=flink-
> > > connectors.git;a=summary to "flink-libraries"
> > > 2. Ask INFRA to set up GitHub and travis integration for
> > "flink-libraries"
> > > 3. Put the code of "flink-ml", "flink-gelly", "flink-python",
> > "flink-cep",
> > > "flink-scala-shell", "flink-storm" into the new repository. (I decided
> > > against moving flink-contrib there, because rocksdb is in the contrib
> > > module, for flink-table, I'm undecided, but I kept it in the main repo
> > > because its probably going to interact more with the core code in the
> > > future)
> > > I try to preserve the history of those modules when splitting them into
> > the
> > > new repo
> > > 4. I'll close all pull requests against those modules in the main repo.
> > > 5. I'll set up a minimal documentation page for the library repository,
> > > similar to the main documentation.
> > > 6. I'll update the documentation build process to build both
> > documentations
> > > & link them to each other
> > > 7. I'll update the nightly deployment process to include both
> > repositories
> > > 8. I'll update the release script to create the Flink release out of
> both
> > > repositories. In order to put the libraries into the opt/ dir of the
> > > release, I'll need to change the build of "flink-dist" so that it first
> > > builds flink core, then the libraries and then the core again with the
> > > libraries as an additional dependency.
> > >
> > > The main question for the community is: do you agree with point 3 ?
> Would
> > > you like to include more or less?
> > >
> > > I'll start with 1. and 2. tomorrow morning.
> > >
> > >
> > >
> > > On Wed, Mar 15, 2017 at 1:48 PM, Till Rohrmann <trohrmann@apache.org>
> > wrote:
> > >
> > >> In theory we could have a merging bot which solves the problem of the
> > >> "commit window". Once the PR passes all tests and has enough +1s, the
> > bot
> > >> could do the merging and, thus, it effectively linearizes the merge
> > >> process.
> > >>
> > >> I think the second point is actually a disadvantage because there is
> not
> > >> such an immediate incentive/pressure to fix the broken module if it
> > lives
> > >> in a separate repository. Furthermore, breaking API changes in the
> core
> > >> will most likely go unnoticed for some time in other modules which are
> > not
> > >> developed so actively. In the worst case these things will only be
> > noticed
> > >> when we try to make a release.
> > >>
> > >> But I also agree that we are not Google and we don't have the
> > capacities to
> > >> maintain such a smooth a build process that we can keep all the code
> in
> > a
> > >> single repository.
> > >>
> > >> I looked a bit into Gradle and as far as I can tell it offers some
> nice
> > >> features wrt incrementally building projects. This would be beneficial
> > for
> > >> local development but it would not solve our build time problems on
> > Travis.
> > >> Gradle intends to introduce a task result cache which allows to reuse
> > >> results across builds. This could help when building on Travis,
> > however, it
> > >> is not yet fully implemented. Moreover, migrating from Maven to Gradle
> > >> won't come for free (there's simply no free lunch out there) and we
> > might
> > >> risk to introduce new bugs. Therefore, I would vote to split the
> > repository
> > >> in order to mitigate our current problems with Travis and the build
> > time in
> > >> general. Whether to use a different build system or not can then be
> > >> discussed as an orthogonal question.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Tue, Mar 14, 2017 at 8:05 PM, Stephan Ewen <sewen@apache.org>
> wrote:
> > >>
> > >>> Some other thoughts on how repository split would help. I am not sure
> > for
> > >>> all of them, so please comment:
> > >>>
> > >>>  - There is less competition for a "commit window". It happens a lot
> > >>> already that you run all tests and want to commit, but there was a
> > commit
> > >>> in the meantime. You rebase, need to re-test, again commit in the
> > >> meantime.
> > >>>    For a "linear" commit history, this may become a bottleneck
> > >> eventually
> > >>> as well.
> > >>>
> > >>>  - There is less risk of broken master. If one repository/modules
> > breaks
> > >>> its master, the others can still continue.
> > >>>
> > >>> Stephan
> > >>>
> > >>>
> > >>> On Fri, Mar 10, 2017 at 12:20 PM, Till Rohrmann <
> trohrmann@apache.org>
> > >>> wrote:
> > >>>
> > >>>> Thanks for all your input. In order to wrap the discussion up I'd
> like
> > >> to
> > >>>> summarize the mentioned points:
> > >>>>
> > >>>> The problem of increasing build times and complexity of the project
> > has
> > >>>> been acknowledged. Ideally we would have everything in one
> repository
> > >>> using
> > >>>> an incremental build tool. Since Maven does not properly support
> this
> > >> we
> > >>>> would have to switch our build tool to something like Gradle, for
> > >>> example.
> > >>>>
> > >>>> Another option is introducing build profiles for different sets
of
> > >>> modules
> > >>>> as well as separating integration and unit tests. The third
> > alternative
> > >>>> would be creating sub-projects with their own repositories. I
> actually
> > >>>> think that these two proposal are not necessarily exclusive and
it
> > >> would
> > >>>> also make sense to have a separation between unit and integration
> > tests
> > >>> if
> > >>>> we split the respository.
> > >>>>
> > >>>> The overall consensus seems to be that we don't want to split the
> > >>> community
> > >>>> and want to keep everything under the same umbrella. I think this
is
> > >> the
> > >>>> right way to go, because otherwise some parts of the project could
> > >> become
> > >>>> second class citizens. Given that and that we continue using Maven,
> I
> > >>> still
> > >>>> think that creating sub-projects for the libraries, for example,
> could
> > >> be
> > >>>> beneficial. A split could reduce the project's complexity and make
> it
> > >>>> potentially easier for libraries to get actively developed. The
main
> > >>>> concern is setting up the build infrastructure to aggregate docs
> from
> > >>>> multiple repositories and making them publicly available.
> > >>>>
> > >>>> Since I started this thread and I would really like to see Flink's
> ML
> > >>>> library being revived again, I'd volunteer investigating first
> whether
> > >> it
> > >>>> is doable establishing a proper incremental build for Flink. If
that
> > >>> should
> > >>>> not be possible, I will look into splitting the repository, first
> only
> > >>> for
> > >>>> the libraries. I'll share my results with the community once I'm
> done
> > >>> with
> > >>>> the investigation.
> > >>>>
> > >>>> Cheers,
> > >>>> Till
> > >>>>
> > >>>> On Fri, Feb 24, 2017 at 3:50 PM, Robert Metzger <
> rmetzger@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>>> @Jin Mingjian: You can not use the paid travis version for
open
> > >> source
> > >>>>> projects. It only works for private repositories (at least
back
> then
> > >>> when
> > >>>>> we've asked them about that).
> > >>>>>
> > >>>>> @Stephan: I don't think that incremental builds will be available
> > >> with
> > >>>>> Maven anytime soon.
> > >>>>>
> > >>>>> I agree that we need to fix the build time issue on Travis.
I've
> > >>> recently
> > >>>>> pushed a commit to use now three instead of two test groups.
> > >>>>> But I don't think that this is feasible long-term solution.
> > >>>>>
> > >>>>> If this discussion is only about reducing the build and test
time,
> > >>>>> introducing build profiles for different components as Aljoscha
> > >>> suggested
> > >>>>> would solve the problem Till mentioned.
> > >>>>> Also, if we decide that travis is not a good tool anymore for
the
> > >>>> testing,
> > >>>>> I guess we can find a different solution. There are now competitors
> > >> to
> > >>>>> Travis that might be willing to offer a paid plan for an open
> source
> > >>>>> project, or we set up our own infra on a server sponsored by
one of
> > >> the
> > >>>>> contributing companies.
> > >>>>> If we want to solve "community issues" with the change as well,
> then
> > >> I
> > >>>>> think its work the effort of splitting up Flink into different
> > >>>>> repositories.
> > >>>>>
> > >>>>> Splitting up repositories is not a trivial task in my opinion.
As
> > >>> others
> > >>>>> have mentioned before, we need to consider the following things:
> > >>>>> - How are we doing to build the documentation? Ideally every
repo
> > >>> should
> > >>>>> contain its docs, so we would need to pull them together when
> > >> building
> > >>>> the
> > >>>>> main docs.
> > >>>>> - How do organize the dependencies? If we have library repository
> > >>> depend
> > >>>> on
> > >>>>> snapshot Flink versions, we need to make sure that the snapshot
> > >>>> deployment
> > >>>>> always works. This also means that people working on a library
> > >>> repository
> > >>>>> will pull from snapshot OR need to build first locally.
> > >>>>> - We need to update the release scripts
> > >>>>>
> > >>>>> If we commit to do these changes, we need to assign at least
one
> > >>>> committer
> > >>>>> (yes, in this case we need somebody who can commit, for example
for
> > >>>>> updating the buildbot stuff) who volunteers to do the change.
> > >>>>> I've done a lot of infrastructure work in the past, but I'm
> currently
> > >>>>> pretty booked with many other things, so I don't realistically
see
> > >>> myself
> > >>>>> doing that. Max who used to work on these things is taking
some
> time
> > >>> off.
> > >>>>> I think we need, best case 3 days for the change, worst case
5
> days.
> > >>> The
> > >>>>> problem is that there are no "unit tests" for the infra stuff,
so
> > >> many
> > >>>>> things are "trial and error" (like Apache's buildbot, our release
> > >>>> scripts,
> > >>>>> the doc scripts, maven stuff, nightly builds).
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Thu, Feb 23, 2017 at 1:33 PM, Stephan Ewen <sewen@apache.org>
> > >>> wrote:
> > >>>>>
> > >>>>>> If we can get a incremental builds to work, that would
actually be
> > >>> the
> > >>>>>> preferred solution in my opinion.
> > >>>>>>
> > >>>>>> Many companies have invested heavily in making a "single
> > >> repository"
> > >>>> code
> > >>>>>> base work, because it has the advantage of not having to
> > >>> update/publish
> > >>>>>> several repositories first.
> > >>>>>> However, the strong prerequisite for that is an incremental
build
> > >>>> system
> > >>>>>> that builds only (fine grained) what it has to build. I
am not
> sure
> > >>> how
> > >>>>> we
> > >>>>>> could make that work
> > >>>>>> with Maven and Travis...
> > >>>>>>
> > >>>>>> On Wed, Feb 22, 2017 at 10:42 PM, Greg Hogan <code@greghogan.com>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> An additional option for reducing time to build and
test is
> > >>> parallel
> > >>>>>>> execution. This would help users more than on TravisCI
since
> > >> we're
> > >>>>>>> generally running on multi-core machines rather than
VM slices.
> > >>>>>>>
> > >>>>>>> Is the idea that each user would only check out the
modules that
> > >> he
> > >>>> or
> > >>>>>> she
> > >>>>>>> is developing with? For example, if a developer is
not working on
> > >>>>>>> flink-mesos or flink-yarn then the "flink-deploy" module
would
> > >> not
> > >>> be
> > >>>>>> clone
> > >>>>>>> to their filesystem?
> > >>>>>>>
> > >>>>>>> We can run a TravisCI nightly build on each repo to
validate
> > >>> against
> > >>>>> API
> > >>>>>>> changes.
> > >>>>>>>
> > >>>>>>> Greg
> > >>>>>>>
> > >>>>>>> On Wed, Feb 22, 2017 at 12:24 PM, Fabian Hueske <
> > >> fhueske@gmail.com
> > >>>>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi everybody,
> > >>>>>>>>
> > >>>>>>>> I think this should be a discussion about the benefits
and
> > >>>> drawbacks
> > >>>>> of
> > >>>>>>>> separating the code into distinct repositories
from a
> > >> development
> > >>>>> point
> > >>>>>>> of
> > >>>>>>>> view.
> > >>>>>>>> So I agree with Stephan that we should not divide
the community
> > >>> by
> > >>>>>>> creating
> > >>>>>>>> separate groups of committers.
> > >>>>>>>> Also the discussion about independent releases
is not be
> > >> strictly
> > >>>>>> related
> > >>>>>>>> to the decision, IMO.
> > >>>>>>>>
> > >>>>>>>> I see a few pros and cons for splitting the code
base into
> > >>> separate
> > >>>>>>>> repositories which (I think) haven't been mentioned
before:
> > >>>>>>>> pros:
> > >>>>>>>> - IDE setup will be leaner. It is not necessary
to compile the
> > >>>> whole
> > >>>>>> code
> > >>>>>>>> base to run a test after switching a branch.
> > >>>>>>>> cons:
> > >>>>>>>> - developing libraries features that require changes
in the
> > >> core
> > >>> /
> > >>>>> APIs
> > >>>>>>>> become more time consuming due to back-and-forth
between code
> > >>>> bases.
> > >>>>>>>> However, I think this is not very often the case.
> > >>>>>>>>
> > >>>>>>>> Aljoscha has good points as well. Many of the build
issues
> > >> could
> > >>> be
> > >>>>>>> solved
> > >>>>>>>> by different build profiles and configurations.
> > >>>>>>>>
> > >>>>>>>> Best, Fabian
> > >>>>>>>>
> > >>>>>>>> 2017-02-22 14:59 GMT+01:00 Gábor Hermann <
> > >> mail@gaborhermann.com
> > >>>> :
> > >>>>>>>>
> > >>>>>>>>> @Stephan:
> > >>>>>>>>>
> > >>>>>>>>> Although I tried to raise some issues about
splitting
> > >>> committers,
> > >>>>> I'm
> > >>>>>>>>> still strongly in favor of some kind of restructuring.
We
> > >> just
> > >>>> have
> > >>>>>> to
> > >>>>>>> be
> > >>>>>>>>> conscious about the disadvantages.
> > >>>>>>>>>
> > >>>>>>>>> Not splitting the committers could leave the
libraries in the
> > >>>> same
> > >>>>>>>>> stalling status, described by Till. Of course,
dedicating
> > >>> current
> > >>>>>>>>> committers as shepherds of the libraries could
easily resolve
> > >>> the
> > >>>>>>> issue.
> > >>>>>>>>> But that requires time from current committers.
It seems like
> > >>>>>>> trade-offs
> > >>>>>>>>> between code quality, speed of development,
and committer
> > >>>> efforts.
> > >>>>>>>>>
> > >>>>>>>>> From what I see in the discussion about ML,
there are many
> > >>> people
> > >>>>>>> willing
> > >>>>>>>>> to contribute as well as production use-cases.
This means we
> > >>>> could
> > >>>>>> and
> > >>>>>>>>> should move forward. However, the development
speed is
> > >>>>> significantly
> > >>>>>>>> slowed
> > >>>>>>>>> down by stalling PRs. The proposal for contributors
helping
> > >> the
> > >>>>>> review
> > >>>>>>>>> process did not really work out so far. In
my opinion, either
> > >>>> code
> > >>>>>>>> quality
> > >>>>>>>>> (by more easily accepting new committers) or
some committer
> > >>> time
> > >>>>>>>>> (reviewing/merging) should be sacrificed to
move forward. As
> > >>> Till
> > >>>>> has
> > >>>>>>>>> indicated, it would be shameful if we let this
contribution
> > >>>> effort
> > >>>>>> die.
> > >>>>>>>>>
> > >>>>>>>>> Cheers,
> > >>>>>>>>> Gabor
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message