flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7047) Reorganize build profiles
Date Tue, 04 Jul 2017 15:30:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073822#comment-16073822

ASF GitHub Bot commented on FLINK-7047:

Github user aljoscha commented on the issue:

    I think we should be able to do this as a two-step process without changing the poms:
    mvn install -pl flink-tests,flink-foo-bar -am -DskipTests -Dmaven.javadoc.skip=true
    mvn verify -pl flink-tests,flink-foo-bar -Dmaven.javadoc.skip=true
    The first line will make sure that we have all the dependencies built, the second one
only executes the tests in the given submodules. We cannot do it as one step because running
`mvn verify -pl flink-tests,... -am` would run the tests also on all dependencies of `flink-test,...`.
    However, it seems we can't do this because the second incantation fails with 
    [WARNING] Some problems were encountered while building the effective model for org.apache.flink:flink-scala-shell_2.11:jar:1.4-SNAPSHOT
    [WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.flink:flink-scala-shell_${scala.binary.version}:[unknown-version],
/Users/aljoscha/Dev/flink/flink-scala-shell/pom.xml, line 30, column 14
    that is including the Scala version in our artifact IDs broke this.

> Reorganize build profiles
> -------------------------
>                 Key: FLINK-7047
>                 URL: https://issues.apache.org/jira/browse/FLINK-7047
>             Project: Flink
>          Issue Type: Improvement
>          Components: Tests, Travis
>    Affects Versions: 1.4.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>             Fix For: 1.4.0
> With the current build times once again hitting the timeout it is time to revisit our
> The current approach of splitting all tests by name, while easy to maintain or extend,
has the big disadvantage that it's fairly binary in regards to the timeout: either we're below
the timeout and all builds pass, or we're above and the entire merging process stalls. Furthermore,
it requires all modules to be compiled.
> I propose a different approach by which we bundle several modules, only execute the tests
of these modules and skip the compilation of some modules that are not required for these
> 5 groups are my current suggestion, which will result in 10 build profiles total.
> The groups are:
> # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb
> # *libraries* - flink-libraries and flink-storm
> # *connectors* - flink-connectors, flink-connector-wikiedits, flink-tweet-inputformat
> # *tests* - flink-tests
> # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, flink-dist
> To not increase the total number of profiles to ridiculous numbers i also propose to
only test against 2 combinations of jdk+hadoop+scala:
> # oraclejdk8 + hadoop 2.8.0 + scala 2.11
> # openjdk7 + hadoop 2.4.1 + scala 2.10
> My current estimate is that this will cause profiles to take at most 40 minutes.

This message was sent by Atlassian JIRA

View raw message