Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BCB9A200BC6 for ; Sun, 20 Nov 2016 17:11:49 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BB200160B07; Sun, 20 Nov 2016 16:11:49 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DC1F2160AFE for ; Sun, 20 Nov 2016 17:11:48 +0100 (CET) Received: (qmail 70279 invoked by uid 500); 20 Nov 2016 16:11:48 -0000 Mailing-List: contact commits-help@zeppelin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zeppelin.apache.org Delivered-To: mailing list commits@zeppelin.apache.org Received: (qmail 70269 invoked by uid 99); 20 Nov 2016 16:11:48 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Nov 2016 16:11:48 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E9771E3910; Sun, 20 Nov 2016 16:11:47 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: moon@apache.org To: commits@zeppelin.apache.org Message-Id: <16d89c1fd68d4c74908e4a7804712271@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: zeppelin git commit: [ZEPPELIN-1673] Reduce CI log size and make it faster Date: Sun, 20 Nov 2016 16:11:47 +0000 (UTC) archived-at: Sun, 20 Nov 2016 16:11:49 -0000 Repository: zeppelin Updated Branches: refs/heads/master 58d37edc8 -> 8c087c1b8 [ZEPPELIN-1673] Reduce CI log size and make it faster ### What is this PR for? #### 1. APT is now cached (saving almost **15 secs** per build) (before) screen shot 2016-11-17 at 11 39 39 am

(after)

#### 2. R packages are now cached (almost **250 secs per** build) (before) screen shot 2016-11-17 at 12 04 20 pm

(after)

#### 3. Log size is reduced (at least 30%, removing useless maven info logs) actually, we reduced even more because we cached apt and R :) ```bash // (before result) https://api.travis-ci.org/jobs/176574582/log.txt?deansi=true // (before build) https://travis-ci.org/apache/zeppelin/builds/176574580 // (after result) https://api.travis-ci.org/jobs/176576354/log.txt?deansi=true // (after build) https://travis-ci.org/apache/zeppelin/builds/176576351 $ ls build8414_profile2.log build8416_profile2.log $ ls -alh total 10232 drwxr-xr-x 4 lambda staff 136B Nov 17 12:47 . drwxr-xr-x 13 lambda staff 442B Nov 17 02:49 .. -rw-r--r-- 1 lambda staff 3.0M Nov 17 12:46 build8414_profile2.log -rw-r--r-- 1 lambda staff 2.0M Nov 17 12:47 build8416_profile2.log $ cat build8414_profile2.log | grep Download | wc -l 7186 $ cat build8416_profile2.log | grep Download | wc -l 6 ``` #### Long Motivations Sometimes build fails due to log size for example, ``` Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in org.apache.zeppelin.ticket.TicketContainerTest Running org.apache.zeppelin.security.SecurityUtilsTest The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over). The job has been terminated ``` we can prevent this kind of failure by removing useless maven log such as ``` [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.pom (13 KB at 1233.1 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.jar [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.jar (112 KB at 4858.2 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.pom [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.pom (8 KB at 627.2 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.jar [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.jar (31 KB at 2048.3 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom (12 KB at 860.4 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar (150 KB at 5342.7 KB/sec) ``` ```sh $ CI cat 176445046.log | grep Download | wc -l 7044 $ CI cat 176445046.log | wc -l 38046 $ CI cat 176445046.log| grep -v Download > filtered.log $ CI ls -alh total 7.7M drwxr-xr-x 4 lambda staff 136 Nov 17 02:50 . drwxr-xr-x 13 lambda staff 442 Nov 17 02:49 .. -rw-r--r-- 1 lambda staff 4.3M Nov 17 02:49 176445046.log -rw-r--r-- 1 lambda staff 3.4M Nov 17 02:50 filtered.log ``` ### What type of PR is it? [Improvement] ### What is the Jira issue? [ZEPPELIN-1673](https://issues.apache.org/jira/browse/ZEPPELIN-1673) ### How should this be tested? Checkout CI log and see if there are `Downloading` `Downloaded` log (see also https://api.travis-ci.org/jobs/176445045/log.txt?deansi=true) ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <1amb4a@gmail.com> Closes #1648 from 1ambda/feat-improve-ci and squashes the following commits: 58e14a9 [1ambda] test: R cache 6e20282 [1ambda] chore: Cache R packages 756f546 [1ambda] feat: Add apt cache f404bb3 [1ambda] chore: Set maven log level to warn 7d7ae0b [1ambda] chore: Test mvn version 98452a7 [1ambda] fix: set quite flag to mvn command 1b9ca09 [1ambda] fix: travis cache dir f6e0ca0 [1ambda] fix: Add mvn option a9bb56f [1ambda] fix: Add quote to global env 06c006e [1ambda] fix: Disable download INFO log Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/8c087c1b Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/8c087c1b Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/8c087c1b Branch: refs/heads/master Commit: 8c087c1b8a44a614e56b1e58734b4d60d04d32c4 Parents: 58d37ed Author: 1ambda <1amb4a@gmail.com> Authored: Thu Nov 17 12:03:26 2016 +0900 Committer: Lee moon soo Committed: Sun Nov 20 08:11:43 2016 -0800 ---------------------------------------------------------------------- .travis.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/8c087c1b/.travis.yml ---------------------------------------------------------------------- diff --git a/.travis.yml b/.travis.yml index 641b540..1cb2e1b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -18,9 +18,11 @@ language: java sudo: false cache: + apt: true directories: - .spark-dist - - ${HOME}/.m2/repository/.cache/maven-download-plugin + - ${HOME}/.m2 + - ${HOME}/R - .node_modules addons: @@ -81,7 +83,7 @@ before_script: - tail conf/zeppelin-env.sh script: - - mvn $TEST_FLAG $PROFILE -B $TEST_PROJECTS + - mvn -Dorg.slf4j.simpleLogger.defaultLogLevel=warn $TEST_FLAG $PROFILE -B $TEST_PROJECTS - rm -rf .node_modules; cp -r zeppelin-web/node_modules .node_modules after_success: