beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukasz Cwik <lc...@google.com>
Subject Re: Gradle migration fixit: April 3
Date Mon, 02 Apr 2018 20:02:16 GMT
I opened up PR/4992[1] to disable all maven based Jenkins jobs and leave
them around so that people working on the migration can update them to be
Gradle based. This also ensures that there is plenty of Jenkins capacity
for testing migration to Gradle.

It is important for Committers to be aware that this is being disabled and
to be careful about merging non Gradle migration fixit tasks.

1: https://github.com/apache/beam/pull/4992

On Mon, Apr 2, 2018 at 10:03 AM Lukasz Cwik <lcwik@google.com> wrote:

> Aljoscha, to add things to be shaded they need to be part of the "compile"
> configuration and you also need to tell the shadow plugin to add these and
> how to relocate them like this:
>
> https://github.com/apache/beam/blob/cc2022157865d44e8dba070a766de78853c508bc/sdks/java/core/build.gradle#L77
>
> Romain, the subtasks of BEAM-3249 seem to be centered around:
> 1) Migrating Jenkins
> 2) Making the artifacts/release (shading, jars, javadoc all fall under
> this)
>
> I would defer things that are optional like "ensure concurrency is not
> hardcoded" till after the migration.
>
> On Sat, Mar 31, 2018 at 2:28 AM Romain Manni-Bucau <rmannibucau@gmail.com>
> wrote:
>
>> Thanks for the doc Lukasz
>>
>> So we do 3 groups?
>>
>> 1. Jenkins
>> 2. Release
>> 3. Outputs/behavior (shades, jars, javadoc, ensure concurrency is not
>> hardcoded, ...)
>>
>> ?
>>
>> Happy to be in 3 or 2 (by preference) if you agree.
>>
>> Le 31 mars 2018 10:54, "Aljoscha Krettek" <aljoscha@apache.org> a écrit :
>>
>>> Thanks Luke, that was helpful!
>>>
>>> I've been playing around with gradle a bit and now have a question: By
>>> default the shadow plugin will include dependencies that are "compile" or
>>> "runtime" (compile dependencies are by default also runtime dependencies)
>>> into the shaded jar. Is that correct? The reason our build doesn't do that
>>> is that we have custom include/exclude rules that prevent that from
>>> happening. I just noticed that when I change the Flink dependencies from
>>> shadow to compile in the Flink Runner build file the shaded jar will not
>>> contain the Flink classes.
>>>
>>> Btw, I'm not saying that I want to include the Flink classes, I was just
>>> messing around with things to get a feeling for how things work.
>>>
>>> On 31. Mar 2018, at 00:15, Lukasz Cwik <lcwik@google.com> wrote:
>>>
>>> I have started a doc[1] containing a Gradle primer to help people be
>>> more productive during the fixit day. Feel free to add/update comments and
>>> content.
>>>
>>> 1:
>>> https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing
>>>
>>> On Fri, Mar 30, 2018 at 8:33 AM Reuven Lax <relax@google.com> wrote:
>>>
>>>> Here is the Kanban board tracking all the current tasks. It looks like
>>>> 7 of them have already been closed over the past two days, so we're down
to
>>>> 19!
>>>>
>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>>>
>>>> On Thu, Mar 29, 2018 at 11:21 PM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Yep - sorry if it was unclear. I know linux distro often do it (never
>>>>> understood why though).
>>>>>
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>> 2018-03-30 7:54 GMT+02:00 Reuven Lax <relax@google.com>:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 29, 2018 at 10:28 PM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> It was more about the drop of poms (same case as you for dataflow).
>>>>>>>
>>>>>>
>>>>>> Ah - you're worried that some external users are building directly
>>>>>> from the poms rather than using the published artifact.
>>>>>>
>>>>>> I think this is a valid concern, and I agree we should announce on
>>>>>> users@ before (probably some time before) deleting the poms  .
>>>>>>
>>>>>>
>>>>>>> On that there is a missing but highly important task: gradle
to mvn
>>>>>>> descriptors. All the one I saw were corrupted poms so we must
take care of
>>>>>>> that as part of the release work (I can work on it on the 3rd
if you want).
>>>>>>>
>>>>>>> Le 29 mars 2018 23:36, "Reuven Lax" <relax@google.com>
a écrit :
>>>>>>>
>>>>>>>> I don't mind notifying users@, but this does seem more interesting
>>>>>>>> for dev@. We will continue to publish Maven artifacts from
our
>>>>>>>> Gradle build, so users are still free to use either Maven
or Gradle.
>>>>>>>>
>>>>>>>> That being said, if this is interesting to users@ we can
notify
>>>>>>>> them as well.
>>>>>>>>
>>>>>>>> Reuven
>>>>>>>>
>>>>>>>> On Thu, Mar 29, 2018 at 1:45 PM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 29 mars 2018 21:20, "Reuven Lax" <relax@google.com>
a écrit :
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 29, 2018 at 12:17 PM Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Le 29 mars 2018 20:35, "Reuven Lax" <relax@google.com>
a écrit :
>>>>>>>>>>
>>>>>>>>>> 1. As Luke already mentioned, we should first have
a subgoal of
>>>>>>>>>> the Gradle jenkins jobs being equivalent to the Maven
jobs. Hopefully
>>>>>>>>>> toward the end of the day, we'll make this change.
>>>>>>>>>>
>>>>>>>>>> 2. Let's see how much progress we make on the third.
There is a
>>>>>>>>>> side problem we have here at Google - we have an
internal product called
>>>>>>>>>> Dataflow built on Beam, and the Dataflow build still
depends on those pom
>>>>>>>>>> files. I would request leaving the pom files around
just a little bit
>>>>>>>>>> longer even if Beam no longer needs them, just so
that we don't breakd
>>>>>>>>>> Dataflow (and I think we would do this for any community
members with a
>>>>>>>>>> similar issue). We will prioritize moving Dataflow
ASAP, it's just that the
>>>>>>>>>> people who will do so will also be in the April 3
fixit so it can't happen
>>>>>>>>>> until after. I think the delay should only be one
or two weeks to delete
>>>>>>>>>> the poms (assuming that Beam is ready at the end
of the day).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you try to put a date of that and we will communicate
on it
>>>>>>>>>> publicly if anyone else does (i dont think but i
didnt expect you to do it
>>>>>>>>>> too ;)).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What do you mean by communicate publicly? The dev list
is already
>>>>>>>>> cced here :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Misses a (not too far) date and users@ pby to be safe
;)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 29, 2018 at 5:20 AM Romain Manni-Bucau
<
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Reuven, a few questions:
>>>>>>>>>>>
>>>>>>>>>>> 1. any inputs on how we can work on the jenkins
part? Do we test
>>>>>>>>>>> it live wiht "fake" PRs?
>>>>>>>>>>> 2. What's the rational to not start by deleting
the poms? Sounds
>>>>>>>>>>> like it will be a day working on gradle and on
the 4th we'll be back on
>>>>>>>>>>> maven
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Romain Manni-Bucau
>>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau>
|  Blog
>>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old
Blog
>>>>>>>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau>
| Book
>>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>>>
>>>>>>>>>>> 2018-03-29 4:46 GMT+02:00 Reuven Lax <relax@google.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> Last week we discussed having a "fixit" day
for Gradle, and I
>>>>>>>>>>>> volunteered to organize it. A number of people
volunteered to help, from
>>>>>>>>>>>> multiple organization. I'd like to say that
it's great to see such a
>>>>>>>>>>>> diverse set of people volunteering to help
here - this is a great way to
>>>>>>>>>>>> build community! Everyone who explicitly
volunteered is directly cced on
>>>>>>>>>>>> this email, though we'd love for more of
the community to help.
>>>>>>>>>>>>
>>>>>>>>>>>> The agreed upon date is April 3. The top-level
JIRA tracking
>>>>>>>>>>>> this work is
>>>>>>>>>>>> ttps://issues.apache.org/jira/browse/BEAM-3249
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-3249>,
and we
>>>>>>>>>>>> currently have 26 subtasks linked to it.
I've created a Kanban board to
>>>>>>>>>>>> track these issues, which I'll share out
soon. We will use Slack the day of
>>>>>>>>>>>> the fixit for collaboration and for questions.
>>>>>>>>>>>>
>>>>>>>>>>>> Two major goals for this fixit should be
to 1. Remove Maven
>>>>>>>>>>>> runs from our Jenkins executors and 2. to
migrate our release process fully
>>>>>>>>>>>> over to Gradle. A lot of work has already
been done on 1., and we've made
>>>>>>>>>>>> some progress on 2.. Slightly longer-term
the goal is to delete all of the
>>>>>>>>>>>> pom files; I'm not sure we'll get as far
as completely deleting Maven in
>>>>>>>>>>>> one day, but we should get within striking
distance!
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance to everyone who's helping
out!
>>>>>>>>>>>>
>>>>>>>>>>>> Reuven
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>

Mime
View raw message