beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reuven Lax <re...@google.com>
Subject Re: Gradle migration fixit: April 3
Date Wed, 04 Apr 2018 15:32:50 GMT
Yesterday we made progress migrating Python to Gradle. We had PRs (some
still pending) from people around the world (from the US to France to
Poland) helping out.

Unfortunately many of the participants yesterday were unfamiliar with any
of the infrastructure, and needed to be taught how Jenkins, Maven, and
Gradle worked. As a result, the people who did have expertise spent much of
the day assisting and teaching instead of migrating, and so we got less
than expected done.

I 100% agree that we don't want to stall again for months. We here at
Google plan to continue working on this fixit with a goal of finishing (we
will be postponing other work we had planned in order to get this done). I
would love to continue to see the entire community continue to help out
here. Yesterday was slower than expected due to the learning curve ofr
participants, but we should pick up the pace.

Reuven

On Wed, Apr 4, 2018 at 5:52 AM Romain Manni-Bucau <rmannibucau@gmail.com>
wrote:

> Hi guys,
>
> Anyone has global a view of where we are? When I left yesterday we were
> far to be consummable by maven users and my diff was not that useful yet
> but saw JB was on it.
> Do we make any progress on that in the night?
>
> Concrete question is: when can we drop maven as the primarly build tool
> for beam (assuming we don't have that dataflow blocker which is a detail
> for the project)? What I don't want is we start again for 2-3 months of a 2
> build tools cohexistance state. Did we reach that?
>
>
> Technical side question (unrelated to the "conclusion"): do you also have
> sym links issue with gradle? If i dont gradle clean maven doesnt build
> anymore cause of it (ending up in OOME cause it loops on sym links).
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-04-03 18:45 GMT+02:00 Jean-Baptiste Onofré <jb@nanthrax.net>:
>
>> Indeed. +1
>>
>> More we are digging around gradle, more we have stuff to do to be
>> equivalent to
>> Maven.
>>
>> I'm also clearly doing a build time comparison to see if it's worth the
>> effort
>> to actually migrate. Currently, on my box, gradle doesn't seem super fast
>> compared to Maven. Let me dig here.
>>
>> Regards
>> JB
>>
>> On 04/03/2018 06:43 PM, Alexey Romanenko wrote:
>> > In BEAM-3249 <https://issues.apache.org/jira/browse/BEAM-3249> there
>> are several
>> > subtasks related to documentation and source code update in favour of
>> gradle
>> > - BEAM-3520 <https://issues.apache.org/jira/browse/BEAM-3520
>> >, BEAM-3521
>> > <https://issues.apache.org/jira/browse/BEAM-3521>, BEAM-3522
>> > <https://issues.apache.org/jira/browse/BEAM-3522> and BEAM-3939
>> > <https://issues.apache.org/jira/browse/BEAM-3939> . Mostly, it's
>> config snippets
>> > and CLI commands.
>> >
>> > It seems that we need to do these changes in the end of the all process
>> of
>> > migration since there are many things that are not well defined for now
>> and how
>> > they will work with gradle. In the same time, the website and other
>> > documentation should be updated with gradle-related stuff along with
>> complete
>> > switch to gradle in building process to avoid a confusion for users.
>> >
>> > What do you think?
>> >
>> > WBR,
>> > Alexey
>> >
>> >> On 3 Apr 2018, at 18:35, Jean-Baptiste Onofré <jb@nanthrax.net
>> >> <mailto:jb@nanthrax.net>> wrote:
>> >>
>> >> Good idea, isolated in a dedicated file will be easy to switch off.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 04/03/2018 06:32 PM, Romain Manni-Bucau wrote:
>> >>> +1
>> >>>
>> >>> also ensuring each optional task has a switch off flag will be
>> important
>> >>> (currently i cant use my computer while building with gradle which is
>> a big lost
>> >>> of time in a day)
>> >>>
>> >>>
>> >>> Romain Manni-Bucau
>> >>> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>> >>> <https://rmannibucau.metawerx.net/> | Old Blog
>> >>> <http://rmannibucau.wordpress.com> | Github <
>> https://github.com/rmannibucau> |
>> >>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>> >>> <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> >>>
>> >>> 2018-04-03 18:27 GMT+02:00 Jean-Baptiste Onofré <jb@nanthrax.net
>> >>> <mailto:jb@nanthrax.net>
>> >>> <mailto:jb@nanthrax.net>>:
>> >>>
>> >>>    Hi guys,
>> >>>
>> >>>    "Europe located guys" started to work on test and changes on
>> Gradle.
>> >>>
>> >>>    I would like to propose a little change in the gradle structure.
>> >>>
>> >>>    Today, lot of tasks and plugins definition are describe in an
>> unique file.
>> >>> It's
>> >>>    pretty hard to find quickly what we are looking for.
>> >>>
>> >>>    What do you think to describe each "high level" tasks in a
>> dedicated file
>> >>>    located in the gradle folder.
>> >>>
>> >>>    To illustrate this, I created the following PR:
>> >>>
>> >>>    https://github.com/apache/beam/pull/5004
>> >>>    <https://github.com/apache/beam/pull/5004>
>> >>>
>> >>>    It's about publishing the artifacts on Nexus repositories
>> (snapshot or
>> >>> staging).
>> >>>    The PR is not yet fully ready but you can see a publish.gradle
>> dedicated
>> >>> to this
>> >>>    in the gradle folder:
>> >>>
>> >>>
>> https://github.com/apache/beam/pull/5004/files#diff-2634489a13e9ba4c3db8f067716556a4
>> >>>    <
>> https://github.com/apache/beam/pull/5004/files#diff-2634489a13e9ba4c3db8f067716556a4
>> >
>> >>>
>> >>>    We can imagine to have:
>> >>>
>> >>>    - gradle/jacoco.gradle for code coverage
>> >>>    - gradle/rat.gradle for RAT
>> >>>    - gradle/dependencies.gradle for dependencies resolution
>> >>>    ...
>> >>>
>> >>>    Thoughts ?
>> >>>
>> >>>    Regards
>> >>>    JB
>> >>>
>> >>>    On 03/29/2018 04:46 AM, Reuven Lax wrote:
>> >>>> Hi all,
>> >>>>
>> >>>> Last week we discussed having a "fixit" day for Gradle, and I
>> volunteered to
>> >>>> organize it. A number of people volunteered to help, from multiple
>> organization.
>> >>>> I'd like to say that it's great to see such a diverse set of people
>> volunteering
>> >>>> to help here - this is a great way to build community! Everyone
who
>> explicitly
>> >>>> volunteered is directly cced on this email, though we'd love for
>> more of the
>> >>>> community to help.
>> >>>>
>> >>>> The agreed upon date is April 3. The top-level JIRA tracking this
>> work is
>> >>>>
>> >>>> ttps://issues.apache.org/jira/browse/BEAM-3249
>> >>>    <http://issues.apache.org/jira/browse/BEAM-3249>
>> >>>> <https://issues.apache.org/jira/browse/BEAM-3249
>> >>>    <https://issues.apache.org/jira/browse/BEAM-3249>>, and
we
>> currently have 26
>> >>>> subtasks linked to it. I've created a Kanban board to track these
>> issues,
>> >>>    which
>> >>>> I'll share out soon. We will use Slack the day of the fixit for
>> collaboration
>> >>>> and for questions.
>> >>>>
>> >>>>
>> >>>> Two major goals for this fixit should be to 1. Remove Maven runs
>> from our
>> >>>> Jenkins executors and 2. to migrate our release process fully over
to
>> >>>    Gradle. A
>> >>>> lot of work has already been done on 1., and we've made some
>> progress on 2..
>> >>>> Slightly longer-term the goal is to delete all of the pom files;
I'm
>> not sure
>> >>>> we'll get as far as completely deleting Maven in one day, but we
>> should get
>> >>>> within striking distance!
>> >>>>
>> >>>>
>> >>>> Thanks in advance to everyone who's helping out!
>> >>>>
>> >>>>
>> >>>> Reuven
>> >>>>
>> >>>
>> >>>    --
>> >>>    Jean-Baptiste Onofré
>> >>>    jbonofre@apache.org <mailto:jbonofre@apache.org> <mailto:
>> jbonofre@apache.org>
>> >>>    http://blog.nanthrax.net
>> >>>    Talend - http://www.talend.com
>> >>>
>> >>>
>> >>
>> >> --
>> >> Jean-Baptiste Onofré
>> >> jbonofre@apache.org <mailto:jbonofre@apache.org>
>> >> http://blog.nanthrax.net
>> >> Talend - http://www.talend.com
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

Mime
View raw message