beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henning Rohde <hero...@google.com>
Subject Re: Gradle Status: Migrated!
Date Tue, 01 May 2018 16:19:54 GMT
JB - for your comparison, please also omit cross-compiling all the Go
examples because they are only built using Gradle.




On Tue, May 1, 2018 at 8:59 AM Jean-Baptiste Onofré <jb@nanthrax.net> wrote:

> Thanks for the update Kenn, that makes sense.
>
> I'm checking the artifacts generated by Gradle right now.
>
> Regards
> JB
>
> On 01/05/2018 17:42, Kenneth Knowles wrote:
> > Raw execution time for tasks from clean is not the only thing to test. I
> > would say it is not even important. Try these from clean:
> >
> >   - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew
> > :beam-sdks-java-io-mongodb:test
> >   - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl
> > sdks/java/io/mongodb test -am
> >
> > Quick run on my laptop:
> >
> >   - Gradle: 66s (65s then 1s)
> >   - Maven: 317s (173s then 144s)
> >
> > Of course, the mvn command runs a bunch of useless executions AND it is
> > incorrect because it isn't using built jars. That's part of the point -
> > there is no way to do what you want with mvn. Let's try to make a
> > command that avoids useless work and builds the jars:
> >
> >   - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn
> > -pl sdks/java/io/mongodb test) && (each time)
> >
> > That takes 102s the first time and 64s the second time. And that is
> > about the realistic workflow for someone trying to get something done.
> > Even if we touch a file Gradle finishes in 20s. So the best case for mvn
> > is this head-to-head:
> >
> >   - Gradle: 65s + 20s + 20s + 20s + 20s + ...
> >   - Maven: 102s + 64s + 64s + 64s + 64s + ...
> >
> > Kenn
> >
> >
> > On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré <jb@nanthrax.net
> > <mailto:jb@nanthrax.net>> wrote:
> >
> >     Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle
> >     (using
> >     the wrapper). It's maybe related to my environment.
> >
> >     Anyway, I'm doing a complete build review both in term of building
> >     time,
> >     and equivalence (artifacts publishing, test, plugin execution).
> >
> >     I will provide an update soon.
> >
> >     Regards
> >     JB
> >
> >     On 01/05/2018 16:57, Reuven Lax wrote:
> >      > Luke did gather data which showed that on our Jenkins executors
> the
> >      > Gradle build was much faster than the Maven build. Also right now
> we
> >      > have incremental builds turned off, but once we're confident
> >     enough to
> >      > enable them (at least for local development) that will often drop
> >     build
> >      > times a lot.
> >      >
> >      > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré
> >     <jb@nanthrax.net <mailto:jb@nanthrax.net>
> >      > <mailto:jb@nanthrax.net <mailto:jb@nanthrax.net>>> wrote:
> >      >
> >      >     By the way, I'm curious: did someone evaluate the build time
> gap
> >      >     between Maven
> >      >     and Gradle ? One of the main reason to migrate to Gradle was
> >     the inc
> >      >     build and
> >      >     build time. The builds I have launched are quite the same in
> >      >     duration. I will do
> >      >     deeper tests to evaluate the gap.
> >      >
> >      >     Regards
> >      >     JB
> >      >
> >      >     On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> >      >      > Hi Scott,
> >      >      >
> >      >      > thanks for the update! Just a clarification about IO
> >     performance
> >      >     tests: those
> >      >      > were fully migrated in Beam and all task necessary for
> running
> >      >     them are there
> >      >      > but Jenkins jobs still run mvn commands. This is due the
> >     fact that
> >      >      > PerfkitBenchmarker code (which is invoked by Jenkins and
> >      >     constructs the commands
> >      >      > by itself) was not updated yet. This should be finished
> before
> >      >     fully dropping mvn.
> >      >      >
> >      >      > More on that topic here, in
> >      >      > comments: https://issues.apache.org/jira/browse/BEAM-3942
> >      >      > PR changing the commands to gradle is waiting for PerfKit
> >     devs review
> >      >      > here:
> >      >
> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >      >      >
> >      >      > Best regards,
> >      >      >
> >      >      > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
> >      >     <rmannibucau@gmail.com <mailto:rmannibucau@gmail.com>
> >     <mailto:rmannibucau@gmail.com <mailto:rmannibucau@gmail.com>>
> >      >      > <mailto:rmannibucau@gmail.com
> >     <mailto:rmannibucau@gmail.com> <mailto:rmannibucau@gmail.com
> >     <mailto:rmannibucau@gmail.com>>>>:
> >      >      >
> >      >      >     Hi Scott
> >      >      >
> >      >      >     While
> >      >
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> >      >      >
> >      >
> >       <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> >      >      >     open, gradle is a concurrent of maven but maven must
> >     stay the
> >      >     default build
> >      >      >     tool cause gradle breaks users.
> >      >      >
> >      >      >
> >      >      >     Le 1 mai 2018 01:59, "Scott Wegner"
> >     <swegner@google.com <mailto:swegner@google.com>
> >      >     <mailto:swegner@google.com <mailto:swegner@google.com>>
> >      >      >     <mailto:swegner@google.com <mailto:swegner@google.com>
> >     <mailto:swegner@google.com <mailto:swegner@google.com>>>>
a
> >      >     écrit :
> >      >      >
> >      >      >         Many many of you have been hacking diligently on
> the
> >      >     Gradle build, and
> >      >      >         I'm happy to announce that we now have a
> >      >     fully-functioning Gradle build!
> >      >      >         There's been a ton of progress since our last
> >     update [1]:
> >      >      >
> >      >      >         * Improved nightly snapshot release [2]
> >      >      >         * Improve runner quickstarts [5] [11]
> >      >      >         * Python post-commit ported to Gradle [3]
> >      >      >         * Update performance testing framework for Gradle
> >     [4] [12]
> >      >      >         * Generate javadocs from Gradle [6]
> >      >      >         * Update to latest Gradle version [7] [21]
> >      >      >         * Updated documentation [8] [22]
> >      >      >         * Tune CI build resource usage for Jenkins [9] [19]
> >      >      >         * Improve shading of test jars [10] [13] [14]
> >      >      >         * Add 'errorprone' and 'spotless' static analysis
> >     [15] [24]
> >      >      >         * Improve IntelliJ project generation [16] [17]
> >      >      >         * Reduce number of ValidatesRunner tests [18]
> >      >      >         * Update release documentation for Gradle [20]
> >      >      >         * Update docker build scripts for Gradle [23]
> >      >      >
> >      >      >         The build process and Jenkins environment have
> >     stabilized
> >      >     and we've
> >      >      >         resolved migration blockers. The final step is to
> use
> >      >     Gradle to produce
> >      >      >         an official release. The release documentation has
> >     been
> >      >     updated for
> >      >      >         Gradle and I recommend we use these docs for the
> 2.5.0
> >      >     release. Assuming
> >      >      >         the release goes well, we can declare the
> >     migration fully
> >      >     validated and
> >      >      >         stop supporting dual build systems.
> >      >      >
> >      >      >         During the migration we identified a number of
> >      >     opportunities to improve
> >      >      >         the build even further. Feel free to grab one of
> the
> >      >     items off of the
> >      >      >         JIRA: BEAM-4045 [24]
> >      >      >
> >      >      >         Thanks again to all those that contributed. This
> has
> >      >     truly been a
> >      >      >         community effort!
> >      >      >
> >      >      >         [1]
> >      >
> >
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> >      >      >
> >      >
> >       <
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> >
> >      >      >         [2] https://github.com/apache/beam/pull/5142
> >      >      >         <https://github.com/apache/beam/pull/5142>
> >      >      >         [3] https://github.com/apache/beam/pull/5146
> >      >      >         <https://github.com/apache/beam/pull/5146>
> >      >      >         [4] https://github.com/apache/beam/pull/5003
> >      >      >         <https://github.com/apache/beam/pull/5003>
> >      >      >         [5] https://github.com/apache/beam/pull/5151
> >      >      >         <https://github.com/apache/beam/pull/5151>
> >      >      >         [6] https://github.com/apache/beam/pull/5121
> >      >      >         <https://github.com/apache/beam/pull/5121>
> >      >      >         [7] https://github.com/apache/beam/pull/5104
> >      >      >         <https://github.com/apache/beam/pull/5104>
> >      >      >         [8] https://github.com/apache/beam/pull/5183
> >      >      >         <https://github.com/apache/beam/pull/5183>
> >      >      >         [9] https://github.com/apache/beam/pull/5171
> >      >      >         <https://github.com/apache/beam/pull/5171>
> >      >      >         [10] https://github.com/apache/beam/pull/5117
> >      >      >         <https://github.com/apache/beam/pull/5117>
> >      >      >         [11] https://github.com/apache/beam/pull/5200
> >      >      >         <https://github.com/apache/beam/pull/5200>
> >      >      >         [12] https://github.com/apache/beam/pull/5051
> >      >      >         <https://github.com/apache/beam/pull/5051>
> >      >      >         [13] https://github.com/apache/beam/pull/4740
> >      >      >         <https://github.com/apache/beam/pull/4740>
> >      >      >         [14] https://github.com/apache/beam/pull/4702
> >      >      >         <https://github.com/apache/beam/pull/4702>
> >      >      >         [15] https://github.com/apache/beam/pull/4701
> >      >      >         <https://github.com/apache/beam/pull/4701>
> >      >      >         [16] https://github.com/apache/beam/pull/4626
> >      >      >         <https://github.com/apache/beam/pull/4626>
> >      >      >         [17] https://github.com/apache/beam/pull/4625
> >      >      >         <https://github.com/apache/beam/pull/4625>
> >      >      >         [18] https://github.com/apache/beam/pull/5193
> >      >      >         <https://github.com/apache/beam/pull/5193>
> >      >      >         [19] https://github.com/apache/beam/pull/5222
> >      >      >         <https://github.com/apache/beam/pull/5222>
> >      >      >         [20] https://github.com/apache/beam/pull/5187
> >      >      >         <https://github.com/apache/beam/pull/5187>
> >      >      >         [21] https://github.com/apache/beam/pull/5217
> >      >      >         <https://github.com/apache/beam/pull/5217>
> >      >      >         [22] https://github.com/apache/beam/pull/5115
> >      >      >         <https://github.com/apache/beam/pull/5115>
> >      >      >         [23] https://github.com/apache/beam/pull/5252
> >      >      >         <https://github.com/apache/beam/pull/5252>
> >      >      >         [24] https://github.com/apache/beam/pull/5161
> >      >      >         <https://github.com/apache/beam/pull/5161>
> >      >      >         [25]
> https://issues.apache.org/jira/browse/BEAM-4045
> >      >      >         <https://issues.apache.org/jira/browse/BEAM-4045>
> >      >      >         --
> >      >      >
> >      >      >
> >      >      >         Got feedback? http://go/swegner-feedback
> >      >      >
> >      >      >
> >      >
> >      >     --
> >      >     Jean-Baptiste Onofré
> >      > jbonofre@apache.org <mailto:jbonofre@apache.org>
> >     <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org>>
> >      > http://blog.nanthrax.net
> >      >     Talend - http://www.talend.com
> >      >
> >
>

Mime
View raw message