From dev-return-17281-archive-asf-public=cust-asf.ponee.io@beam.apache.org Wed Jun 5 02:18:41 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0FA8718064D for ; Wed, 5 Jun 2019 04:18:40 +0200 (CEST) Received: (qmail 27436 invoked by uid 500); 5 Jun 2019 02:18:39 -0000 Mailing-List: contact dev-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list dev@beam.apache.org Received: (qmail 27347 invoked by uid 99); 5 Jun 2019 02:18:39 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jun 2019 02:18:39 +0000 Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id CD3CD8A85 for ; Wed, 5 Jun 2019 02:18:38 +0000 (UTC) Received: by mail-ed1-f47.google.com with SMTP id c26so3443545edt.1 for ; Tue, 04 Jun 2019 19:18:38 -0700 (PDT) X-Gm-Message-State: APjAAAXK0naNk7mrgyNYjKWtingWKR7pn1JQU2uJYcg4BG1ZoXTOdgw0 Qbcv/h5TIFfFjtt6wY8fQ0QjmX1RK60NX92f0wOrwA== X-Google-Smtp-Source: APXvYqzJKIfiU6iBWaRI+xRETdadBoWQV8lboVbBEkTMXzo6XQUFp7Tddw8EsLfATB1COwyJVCg17CUR+/LOP5RFDy0= X-Received: by 2002:a50:a56d:: with SMTP id z42mr18215595edb.241.1559701116987; Tue, 04 Jun 2019 19:18:36 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Kenneth Knowles Date: Tue, 4 Jun 2019 19:18:25 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Removing shading by default within BeamModulePlugin.groovy To: dev Content-Type: multipart/alternative; boundary="000000000000090b2c058a8a3743" --000000000000090b2c058a8a3743 Content-Type: text/plain; charset="UTF-8" Nice! This is a huge step. One thing that showed up in the last big gradle change was needing to check the generated poms. Kenn On Tue, Jun 4, 2019 at 5:07 PM Lukasz Cwik wrote: > Since we have been migrating to using vendoring instead of shading[1] and > due to previous efforts in vendoring[2, 3] I have opened up PR 8762[4] > which migrates all projects that weren't doing anything shading wise to not > perform any shading. This required me to fix up all intra project > dependencies and release publishing. > > The following is a list of all project paths which are still using shading > for some reason: > model/* > sdks/java/core > sdks/java/extensions/kryo > sdks/java/extensions/sql > sdks/java/extensions/sql/jdbc > sdks/java/harness > runners/spark/job-server > runners/direct-java > runners/samza/job-server > runners/google-cloud-dataflow-java/worker > runners/google-cloud-dataflow-java/worker/legacy-worker > runners/google-cloud-dataflow-java/worker/windmill > vendor/* > > Out of the list above, migrating sdks/java/core and runners/direct-java > (in that order) would provide the most benefit to moving away from shading > within our project. Many of the others are either shaded proto classes or > applications (e.g. job-servers, harness, sql jdbc) and either require > shading to be compatible with vendoring or aren't meant to be used as > dependencies. > > Since this is a larger change that cuts across so many projects there is > risk for breakage. I'm looking for people to help test the change and > validate any scenarios that they are specifically interested in. I'm > planning to run several of the postcommits on my PR and check that we can > build a release in addition to any efforts others provide before looking to > have the change merged. > > The following guidance should help those who edit Gradle build files > (after this change is merged): > * For projects that don't perform any shading, those projects have been > migrated to use the default configurations that the Gradle Java plugin > uses[5]. Note that the default configurations we use have been deprecated. > * For projects that depend on another project that isn't shaded, the intra > project configuration has been swapped to use compile / testRuntime instead > of shadow and shadowTest > * Existing projects that are still shaded should use the shadow and > shadowTest configurations as before. > > 1: > https://lists.apache.org/thread.html/4c12db35b40a6d56e170cd6fc8bb0ac4c43a99aa3cb7dbae54176815@%3Cdev.beam.apache.org%3E > 2: > https://lists.apache.org/thread.html/4c12db35b40a6d56e170cd6fc8bb0ac4c43a99aa3cb7dbae54176815@%3Cdev.beam.apache.org%3E > 3: > https://lists.apache.org/thread.html/972b5175641f4eaf7ec92870cc0ff72fa52e6f0bbaccc384a3814e45@%3Cdev.beam.apache.org%3E > 4: https://github.com/apache/beam/pull/8762 > 5: > https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin_and_dependency_management > --000000000000090b2c058a8a3743 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Nice! This is a huge step. One thing that showed up in the= last big gradle change was needing to check the generated poms.

Kenn

On Tue, Jun 4, 2019 at 5:07 PM Lukasz Cwik <lcwik@google.com> wrote:
Since we= have been migrating to using vendoring instead of shading[1] and due to pr= evious efforts in vendoring[2, 3] I have opened up PR 8762[4] which migrate= s all projects that weren't doing anything shading wise to not perform = any shading. This required me to fix up all intra project dependencies and = release publishing.

The following is a list of all= project paths which are still using shading for some reason:
mod= el/*
sdks/java/core
sdks/java/extensions/kryo
=
sdks/java/extensions/sql
sdks/java/extensions/sql/jdbc
=
sdks/java/harness
runners/spark/job-server
runners= /direct-java
runners/samza/job-server
runners/google-cl= oud-dataflow-java/worker
runners/google-cloud-dataflow-java/worke= r/legacy-worker
runners/google-cloud-dataflow-java/worker/win= dmill
vendor/*

Out of the list above= , migrating sdks/java/core and runners/direct-java (in that order) would pr= ovide the most benefit to moving away from shading within our project. Many= of the others are either shaded proto classes or applications (e.g. job-se= rvers, harness, sql jdbc) and either require shading to be compatible with = vendoring or aren't meant to be used as dependencies.

Since this is a larger change that cuts across so many p= rojects there is risk for breakage. I'm looking for people to help test= the change and validate any scenarios that they are specifically intereste= d in. I'm planning to run several of the postcommits on my PR and check= that we can build a release in addition to any efforts others provide befo= re looking to have the change merged.

The followin= g guidance should help those who edit Gradle build files (after this change= is merged):
* For projects that don't perform any shading, t= hose projects have been migrated to use the default configurations that the= Gradle Java plugin uses[5]. Note that the default configurations we use ha= ve been deprecated.
* For projects that depend on another pro= ject that isn't shaded, the intra project configuration has been swappe= d to use compile / testRuntime instead of shadow and shadowTest
*= Existing projects that are still shaded should use the shadow and shadowTe= st configurations as before.

--000000000000090b2c058a8a3743--