beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: Gradle Status: Migrated!
Date Tue, 01 May 2018 15:59:39 GMT
Thanks for the update Kenn, that makes sense.

I'm checking the artifacts generated by Gradle right now.

Regards
JB

On 01/05/2018 17:42, Kenneth Knowles wrote:
> Raw execution time for tasks from clean is not the only thing to test. I 
> would say it is not even important. Try these from clean:
> 
>   - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew 
> :beam-sdks-java-io-mongodb:test
>   - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl 
> sdks/java/io/mongodb test -am
> 
> Quick run on my laptop:
> 
>   - Gradle: 66s (65s then 1s)
>   - Maven: 317s (173s then 144s)
> 
> Of course, the mvn command runs a bunch of useless executions AND it is 
> incorrect because it isn't using built jars. That's part of the point - 
> there is no way to do what you want with mvn. Let's try to make a 
> command that avoids useless work and builds the jars:
> 
>   - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn 
> -pl sdks/java/io/mongodb test) && (each time)
> 
> That takes 102s the first time and 64s the second time. And that is 
> about the realistic workflow for someone trying to get something done. 
> Even if we touch a file Gradle finishes in 20s. So the best case for mvn 
> is this head-to-head:
> 
>   - Gradle: 65s + 20s + 20s + 20s + 20s + ...
>   - Maven: 102s + 64s + 64s + 64s + 64s + ...
> 
> Kenn
> 
> 
> On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré <jb@nanthrax.net 
> <mailto:jb@nanthrax.net>> wrote:
> 
>     Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle
>     (using
>     the wrapper). It's maybe related to my environment.
> 
>     Anyway, I'm doing a complete build review both in term of building
>     time,
>     and equivalence (artifacts publishing, test, plugin execution).
> 
>     I will provide an update soon.
> 
>     Regards
>     JB
> 
>     On 01/05/2018 16:57, Reuven Lax wrote:
>      > Luke did gather data which showed that on our Jenkins executors the
>      > Gradle build was much faster than the Maven build. Also right now we
>      > have incremental builds turned off, but once we're confident
>     enough to
>      > enable them (at least for local development) that will often drop
>     build
>      > times a lot.
>      >
>      > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré
>     <jb@nanthrax.net <mailto:jb@nanthrax.net>
>      > <mailto:jb@nanthrax.net <mailto:jb@nanthrax.net>>> wrote:
>      >
>      >     By the way, I'm curious: did someone evaluate the build time gap
>      >     between Maven
>      >     and Gradle ? One of the main reason to migrate to Gradle was
>     the inc
>      >     build and
>      >     build time. The builds I have launched are quite the same in
>      >     duration. I will do
>      >     deeper tests to evaluate the gap.
>      >
>      >     Regards
>      >     JB
>      >
>      >     On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
>      >      > Hi Scott,
>      >      >
>      >      > thanks for the update! Just a clarification about IO
>     performance
>      >     tests: those
>      >      > were fully migrated in Beam and all task necessary for running
>      >     them are there
>      >      > but Jenkins jobs still run mvn commands. This is due the
>     fact that
>      >      > PerfkitBenchmarker code (which is invoked by Jenkins and
>      >     constructs the commands
>      >      > by itself) was not updated yet. This should be finished before
>      >     fully dropping mvn.
>      >      >
>      >      > More on that topic here, in
>      >      > comments: https://issues.apache.org/jira/browse/BEAM-3942
>      >      > PR changing the commands to gradle is waiting for PerfKit
>     devs review
>      >      > here:
>      > https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
>      >      >
>      >      > Best regards,
>      >      >
>      >      > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
>      >     <rmannibucau@gmail.com <mailto:rmannibucau@gmail.com>
>     <mailto:rmannibucau@gmail.com <mailto:rmannibucau@gmail.com>>
>      >      > <mailto:rmannibucau@gmail.com
>     <mailto:rmannibucau@gmail.com> <mailto:rmannibucau@gmail.com
>     <mailto:rmannibucau@gmail.com>>>>:
>      >      >
>      >      >     Hi Scott
>      >      >
>      >      >     While
>      > https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
>      >      >
>      >     
>       <https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057>
is
>      >      >     open, gradle is a concurrent of maven but maven must
>     stay the
>      >     default build
>      >      >     tool cause gradle breaks users.
>      >      >
>      >      >
>      >      >     Le 1 mai 2018 01:59, "Scott Wegner"
>     <swegner@google.com <mailto:swegner@google.com>
>      >     <mailto:swegner@google.com <mailto:swegner@google.com>>
>      >      >     <mailto:swegner@google.com <mailto:swegner@google.com>
>     <mailto:swegner@google.com <mailto:swegner@google.com>>>> a
>      >     écrit :
>      >      >
>      >      >         Many many of you have been hacking diligently on
the
>      >     Gradle build, and
>      >      >         I'm happy to announce that we now have a
>      >     fully-functioning Gradle build!
>      >      >         There's been a ton of progress since our last
>     update [1]:
>      >      >
>      >      >         * Improved nightly snapshot release [2]
>      >      >         * Improve runner quickstarts [5] [11]
>      >      >         * Python post-commit ported to Gradle [3]
>      >      >         * Update performance testing framework for Gradle
>     [4] [12]
>      >      >         * Generate javadocs from Gradle [6]
>      >      >         * Update to latest Gradle version [7] [21]
>      >      >         * Updated documentation [8] [22]
>      >      >         * Tune CI build resource usage for Jenkins [9] [19]
>      >      >         * Improve shading of test jars [10] [13] [14]
>      >      >         * Add 'errorprone' and 'spotless' static analysis
>     [15] [24]
>      >      >         * Improve IntelliJ project generation [16] [17]
>      >      >         * Reduce number of ValidatesRunner tests [18]
>      >      >         * Update release documentation for Gradle [20]
>      >      >         * Update docker build scripts for Gradle [23]
>      >      >
>      >      >         The build process and Jenkins environment have
>     stabilized
>      >     and we've
>      >      >         resolved migration blockers. The final step is to
use
>      >     Gradle to produce
>      >      >         an official release. The release documentation has
>     been
>      >     updated for
>      >      >         Gradle and I recommend we use these docs for the
2.5.0
>      >     release. Assuming
>      >      >         the release goes well, we can declare the
>     migration fully
>      >     validated and
>      >      >         stop supporting dual build systems.
>      >      >
>      >      >         During the migration we identified a number of
>      >     opportunities to improve
>      >      >         the build even further. Feel free to grab one of
the
>      >     items off of the
>      >      >         JIRA: BEAM-4045 [24]
>      >      >
>      >      >         Thanks again to all those that contributed. This
has
>      >     truly been a
>      >      >         community effort!
>      >      >
>      >      >         [1]
>      >
>     https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
>      >      >
>      >     
>       <https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E>
>      >      >         [2] https://github.com/apache/beam/pull/5142
>      >      >         <https://github.com/apache/beam/pull/5142>
>      >      >         [3] https://github.com/apache/beam/pull/5146
>      >      >         <https://github.com/apache/beam/pull/5146>
>      >      >         [4] https://github.com/apache/beam/pull/5003
>      >      >         <https://github.com/apache/beam/pull/5003>
>      >      >         [5] https://github.com/apache/beam/pull/5151
>      >      >         <https://github.com/apache/beam/pull/5151>
>      >      >         [6] https://github.com/apache/beam/pull/5121
>      >      >         <https://github.com/apache/beam/pull/5121>
>      >      >         [7] https://github.com/apache/beam/pull/5104
>      >      >         <https://github.com/apache/beam/pull/5104>
>      >      >         [8] https://github.com/apache/beam/pull/5183
>      >      >         <https://github.com/apache/beam/pull/5183>
>      >      >         [9] https://github.com/apache/beam/pull/5171
>      >      >         <https://github.com/apache/beam/pull/5171>
>      >      >         [10] https://github.com/apache/beam/pull/5117
>      >      >         <https://github.com/apache/beam/pull/5117>
>      >      >         [11] https://github.com/apache/beam/pull/5200
>      >      >         <https://github.com/apache/beam/pull/5200>
>      >      >         [12] https://github.com/apache/beam/pull/5051
>      >      >         <https://github.com/apache/beam/pull/5051>
>      >      >         [13] https://github.com/apache/beam/pull/4740
>      >      >         <https://github.com/apache/beam/pull/4740>
>      >      >         [14] https://github.com/apache/beam/pull/4702
>      >      >         <https://github.com/apache/beam/pull/4702>
>      >      >         [15] https://github.com/apache/beam/pull/4701
>      >      >         <https://github.com/apache/beam/pull/4701>
>      >      >         [16] https://github.com/apache/beam/pull/4626
>      >      >         <https://github.com/apache/beam/pull/4626>
>      >      >         [17] https://github.com/apache/beam/pull/4625
>      >      >         <https://github.com/apache/beam/pull/4625>
>      >      >         [18] https://github.com/apache/beam/pull/5193
>      >      >         <https://github.com/apache/beam/pull/5193>
>      >      >         [19] https://github.com/apache/beam/pull/5222
>      >      >         <https://github.com/apache/beam/pull/5222>
>      >      >         [20] https://github.com/apache/beam/pull/5187
>      >      >         <https://github.com/apache/beam/pull/5187>
>      >      >         [21] https://github.com/apache/beam/pull/5217
>      >      >         <https://github.com/apache/beam/pull/5217>
>      >      >         [22] https://github.com/apache/beam/pull/5115
>      >      >         <https://github.com/apache/beam/pull/5115>
>      >      >         [23] https://github.com/apache/beam/pull/5252
>      >      >         <https://github.com/apache/beam/pull/5252>
>      >      >         [24] https://github.com/apache/beam/pull/5161
>      >      >         <https://github.com/apache/beam/pull/5161>
>      >      >         [25] https://issues.apache.org/jira/browse/BEAM-4045
>      >      >         <https://issues.apache.org/jira/browse/BEAM-4045>
>      >      >         --
>      >      >
>      >      >
>      >      >         Got feedback? http://go/swegner-feedback
>      >      >
>      >      >
>      >
>      >     --
>      >     Jean-Baptiste Onofré
>      > jbonofre@apache.org <mailto:jbonofre@apache.org>
>     <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org>>
>      > http://blog.nanthrax.net
>      >     Talend - http://www.talend.com
>      >
> 

Mime
View raw message