flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From uce <...@git.apache.org>
Subject [GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Date Mon, 16 Feb 2015 16:34:18 GMT
GitHub user uce opened a pull request:

    https://github.com/apache/flink/pull/407

    [builds] Allow to upload build artifacts to S3

    This PR includes #389.
    
    Currently, logging of Travis builds is disabled as the log files are discarded after a
build and stdout logging is limited in size. With this PR, we can log to a file and have them
uploaded automatically.
    
    1. **Console logging**: For console logging, we can use `MavenForkNumberPrefixLayout`
to prefix the log output with the fork number, which makes it easier to group log output together.
    
      We might consider moving that layout to a different package, though.
    
    ```
    7 > 17:12:23,220 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Deploying test vertex (5/7) (attempt #0) to host3
    7 > 17:12:23,220 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler    
  - Scheduling task {task=test vertex (6/7) - execution #0, sharingUnit=null, locationConstraint=null}
    7 > 17:12:23,220 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler    
  - Unconstrained assignment: test vertex (6/7) --> dbc442af2610e30d3309d96e14fa9d8a @
akka://TestingActorSystem/user/$$a - 5 slots - 1067599825
    7 > 17:12:23,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Deploying test vertex (6/7) (attempt #0) to host1
    7 > 17:12:23,221 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler    
  - Scheduling task {task=test vertex (7/7) - execution #0, sharingUnit=null, locationConstraint=null}
    7 > 17:12:23,222 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler    
  - Unconstrained assignment: test vertex (7/7) --> 8ce93c214951df5a8ad32c912ba0e456 @
akka://TestingActorSystem/user/$$a - 5 slots - 749927456
    7 > 17:12:23,222 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Deploying test vertex (7/7) (attempt #0) to host2
    7 > 17:12:23,223 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
    7 > 17:12:23,223 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Connecting ExecutionJobVertex 126e16c348d4e5aa129104702700bf9e (test vertex) to 0 predecessors.
    10 > 17:12:23,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Deploying v1 (2/2) (attempt #0) to host3
    6 > 17:12:23,236 DEBUG akka.event.EventStream                                     
  - shutting down: StandardOutLogger started
    1 > 17:12:23,239 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Sending out cancel request, to remove task execution from TaskManager.
    1 > 17:12:23,244 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Concurrent canceling/failing of TestVertex (1/1) - execution #0 while deployment was in
progress.
    10 > 17:12:23,266 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
    Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.61 sec - in org.apache.flink.runtime.executiongraph.ExecutionVertexCancelTest
    Running org.apache.flink.runtime.io.network.api.reader.UnionBufferReaderTest
    10 > 17:12:23,269 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Connecting ExecutionJobVertex e8869426e7765e498ad219e3af731729 (test vertex) to 0 predecessors.
    10 > 17:12:23,269 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Scheduling task {task=test vertex (1/2) - execution #0, sharingUnit=null, locationConstraint=null}
    10 > 17:12:23,270 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Local assignment: test vertex (1/2) --> ee9357ffc16d51c4f7522bc479a757ac @ akka://TestingActorSystem/user/$$a
- 1 slots - 2073640037
    10 > 17:12:23,270 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Deploying test vertex (1/2) (attempt #0) to host1
    10 > 17:12:23,270 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Scheduling task {task=test vertex (2/2) - execution #0, sharingUnit=null, locationConstraint=null}
    10 > 17:12:23,271 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Local assignment: test vertex (2/2) --> 6620d9f45b7567e2b147f5761d8060e8 @ akka://TestingActorSystem/user/$$a
- 1 slots - 1864116663
    10 > 17:12:23,271 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Deploying test vertex (2/2) (attempt #0) to host3
    1 > 17:12:23,277 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Deploying TestVertex (1/1) (attempt #0) to localhost
    10 > 17:12:23,277 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
    10 > 17:12:23,278 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Connecting ExecutionJobVertex 533fcd1e05bcce9ec185ccea7085a5aa (test vertex) to 0 predecessors.
    10 > 17:12:23,278 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Scheduling task {task=test vertex (1/2) - execution #0, sharingUnit=null, locationConstraint=null}
    1 > 17:12:23,280 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Sending out cancel request, to remove task execution from TaskManager.
    1 > 17:12:23,280 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Cancel task call did not find task. Probably akka message call race.
    1 > 17:12:23,281 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Concurrent canceling/failing of TestVertex (1/1) - execution #0 while deployment was in
progress.
    10 > 17:12:23,281 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Local assignment: test vertex (1/2) --> b923679116189038c5dbd280d43cfe48 @ akka://TestingActorSystem/user/$$a
- 1 slots - 44559647
    10 > 17:12:23,282 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph    
   - Deploying test vertex (1/2) (attempt #0) to host3
    10 > 17:12:23,282 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Scheduling task {task=test vertex (2/2) - execution #0, sharingUnit=null, locationConstraint=null}
    10 > 17:12:23,283 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler   
   - Local assignment: test vertex (2/2) --> 1fb78d024e2664602ee31c47dd987f6c @ akka://TestingActorSystem/user/$$a
- 1 slots - 1081769770
    ```
      I've added an example log4j properties file to `flink-runtime/tests`.
    
    2. **File logging**: Each forked build sets its own log file (e.g. `1.log` and `2.log`)
if it is enabled (example in `flink-runtime/tests` log4j properties file). If the `UPLOAD_ARTIFACTS`
variable is set in Travis and the S3 credentials have been set for the respective repository,
the files are uploaded.
    
      As an example, see [this build](https://travis-ci.org/uce/incubator-flink/builds/50960309),
which I've stalled on purpose. It upload the jps traces: https://s3.amazonaws.com/flink.a.o.uce.east/travis-artifacts/uce/incubator-flink/210/210.5/jps-traces.out
    
    ---
    
    The used S3 bucket is my private bucket and the upload currently only works for my forked
repository. It is very easy to add more committers for their own repositories/own buckets.
    
    We can finalize the setup for interested committers after review. I still want to add
a minor change to compress the output.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/uce/incubator-flink travis-upload_artifacts

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #407
    
----
commit 7887599e27949f72373232205d15010c18108bc2
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-02-12T11:19:40Z

    [builds] Print Java process stack traces of stalled builds

commit cb17e87b3d47336f50999c2e239d8ec6467e0a59
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-02-16T15:40:49Z

    [builds] Add logger layout to prefix log output with Maven fork number

commit 488946ee273de9c374ec0a87652077c355718cae
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-02-16T15:42:13Z

    [builds] Move .gitignores and set JVM properties when running build

commit 2a942e0980e80201acc954abc20401a3f9c8764c
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-02-16T15:48:07Z

    [builds] Allow to upload build artifacts to S3

commit 5cd3e395cea2e64b76edebb26ec6a517b111fc85
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-02-16T15:56:48Z

    [builds] Fix rat excludes

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message