zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From m...@apache.org
Subject zeppelin git commit: [ZEPPELIN-1673] Reduce CI log size and make it faster
Date Sun, 20 Nov 2016 16:11:47 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/master 58d37edc8 -> 8c087c1b8


[ZEPPELIN-1673] Reduce CI log size and make it faster

### What is this PR for?

#### 1. APT is now cached (saving almost **15 secs** per build)

(before)
<img width="657" alt="screen shot 2016-11-17 at 11 39 39 am" src="https://cloud.githubusercontent.com/assets/4968473/20375553/2b306884-acc3-11e6-995b-045dec268e0a.png">

(after)
<img width="663" alt="screen shot 2016-11-17 at 11 39 51 am" src="https://cloud.githubusercontent.com/assets/4968473/20375557/2ec0c7aa-acc3-11e6-8468-f672be52826e.png">

#### 2. R packages are now cached (almost **250 secs per** build)

(before)
<img width="1131" alt="screen shot 2016-11-17 at 12 04 20 pm" src="https://cloud.githubusercontent.com/assets/4968473/20375573/4c9cbc84-acc3-11e6-9717-ce6a6d9ad360.png">

(after)
<img width="1134" alt="screen shot 2016-11-17 at 12 37 19 pm" src="https://cloud.githubusercontent.com/assets/4968473/20375585/526ccc58-acc3-11e6-95a9-8446a298597a.png">
<img width="340" alt="screen shot 2016-11-17 at 12 37 34 pm" src="https://cloud.githubusercontent.com/assets/4968473/20375588/57227f68-acc3-11e6-8ef7-9a86b125c00d.png">

#### 3. Log size is reduced (at least 30%, removing useless maven info logs)

actually, we reduced even more because we cached apt and R :)

```bash
// (before result) https://api.travis-ci.org/jobs/176574582/log.txt?deansi=true
// (before build) https://travis-ci.org/apache/zeppelin/builds/176574580

// (after result) https://api.travis-ci.org/jobs/176576354/log.txt?deansi=true
// (after build) https://travis-ci.org/apache/zeppelin/builds/176576351

$ ls
build8414_profile2.log build8416_profile2.log

$ ls -alh
total 10232
drwxr-xr-x   4 lambda  staff   136B Nov 17 12:47 .
drwxr-xr-x  13 lambda  staff   442B Nov 17 02:49 ..
-rw-r--r--   1 lambda  staff   3.0M Nov 17 12:46 build8414_profile2.log
-rw-r--r--   1 lambda  staff   2.0M Nov 17 12:47 build8416_profile2.log

$ cat build8414_profile2.log | grep Download | wc -l
    7186
$ cat build8416_profile2.log | grep Download | wc -l
       6
```

#### Long Motivations

Sometimes build fails due to log size for example,

```
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in org.apache.zeppelin.ticket.TicketContainerTest
Running org.apache.zeppelin.security.SecurityUtilsTest

The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising
the same exception over and over).

The job has been terminated
```

we can prevent this kind of failure by removing useless maven log such as

```
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.pom
(13 KB at 1233.1 KB/sec)
[INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.jar
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.jar
(112 KB at 4858.2 KB/sec)
[INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.pom
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.pom
(8 KB at 627.2 KB/sec)
[INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.jar
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.jar
(31 KB at 2048.3 KB/sec)
[INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom
(12 KB at 860.4 KB/sec)
[INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar
[INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar
(150 KB at 5342.7 KB/sec)
```

```sh
$  CI cat 176445046.log | grep Download | wc -l
    7044
$  CI cat 176445046.log | wc -l
   38046
$  CI cat 176445046.log| grep -v Download > filtered.log
$  CI ls -alh
total 7.7M
drwxr-xr-x  4 lambda staff  136 Nov 17 02:50 .
drwxr-xr-x 13 lambda staff  442 Nov 17 02:49 ..
-rw-r--r--  1 lambda staff 4.3M Nov 17 02:49 176445046.log
-rw-r--r--  1 lambda staff 3.4M Nov 17 02:50 filtered.log
```

### What type of PR is it?
[Improvement]

### What is the Jira issue?

[ZEPPELIN-1673](https://issues.apache.org/jira/browse/ZEPPELIN-1673)

### How should this be tested?

Checkout CI log and see if there are `Downloading` `Downloaded` log

(see also https://api.travis-ci.org/jobs/176445045/log.txt?deansi=true)

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO

Author: 1ambda <1amb4a@gmail.com>

Closes #1648 from 1ambda/feat-improve-ci and squashes the following commits:

58e14a9 [1ambda] test: R cache
6e20282 [1ambda] chore: Cache R packages
756f546 [1ambda] feat: Add apt cache
f404bb3 [1ambda] chore: Set maven log level to warn
7d7ae0b [1ambda] chore: Test mvn version
98452a7 [1ambda] fix: set quite flag to mvn command
1b9ca09 [1ambda] fix: travis cache dir
f6e0ca0 [1ambda] fix: Add mvn option
a9bb56f [1ambda] fix: Add quote to global env
06c006e [1ambda] fix: Disable download INFO log


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/8c087c1b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/8c087c1b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/8c087c1b

Branch: refs/heads/master
Commit: 8c087c1b8a44a614e56b1e58734b4d60d04d32c4
Parents: 58d37ed
Author: 1ambda <1amb4a@gmail.com>
Authored: Thu Nov 17 12:03:26 2016 +0900
Committer: Lee moon soo <moon@apache.org>
Committed: Sun Nov 20 08:11:43 2016 -0800

----------------------------------------------------------------------
 .travis.yml | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/8c087c1b/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index 641b540..1cb2e1b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -18,9 +18,11 @@ language: java
 sudo: false
 
 cache:
+  apt: true
   directories:
     - .spark-dist
-    - ${HOME}/.m2/repository/.cache/maven-download-plugin
+    - ${HOME}/.m2
+    - ${HOME}/R
     - .node_modules
 
 addons:
@@ -81,7 +83,7 @@ before_script:
   - tail conf/zeppelin-env.sh
 
 script:
-  - mvn $TEST_FLAG $PROFILE -B $TEST_PROJECTS
+  - mvn -Dorg.slf4j.simpleLogger.defaultLogLevel=warn $TEST_FLAG $PROFILE -B $TEST_PROJECTS
   - rm -rf .node_modules; cp -r zeppelin-web/node_modules .node_modules
 
 after_success:


Mime
View raw message