From dev-return-3914-archive-asf-public=cust-asf.ponee.io@singa.apache.org Tue Nov 19 22:24:56 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 35F61180638 for ; Tue, 19 Nov 2019 23:24:56 +0100 (CET) Received: (qmail 17702 invoked by uid 500); 19 Nov 2019 22:24:55 -0000 Mailing-List: contact dev-help@singa.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@singa.apache.org Delivered-To: mailing list dev@singa.apache.org Received: (qmail 17690 invoked by uid 99); 19 Nov 2019 22:24:55 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.42) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Nov 2019 22:24:55 +0000 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 32AB84FB4 for ; Tue, 19 Nov 2019 22:24:55 +0000 (UTC) Received: by mail-ed1-f45.google.com with SMTP id w6so18533891edx.10 for ; Tue, 19 Nov 2019 14:24:55 -0800 (PST) X-Gm-Message-State: APjAAAU3qIkHzMzEcyLCt1x1M1+4mzDntimF43fP0yV38TqZX6AzDpr0 QwABmHdczWeZXJ86AQkuRerhqGggMPXpAZ1G8aA= X-Google-Smtp-Source: APXvYqzSjNPM5ya1JpuXyNuUIACG4KNzvk3qHZId3zampqwOUmyyGaPpUTpd0T3usx88Q59QhlH3kuiB+0g9EBCzrug= X-Received: by 2002:a17:906:390a:: with SMTP id f10mr46626eje.310.1574202294387; Tue, 19 Nov 2019 14:24:54 -0800 (PST) MIME-Version: 1.0 From: Moaz Reyad Date: Tue, 19 Nov 2019 23:24:43 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: [DISCUSS] Add code coverage analysis To: dev@singa.apache.org Content-Type: multipart/alternative; boundary="0000000000008fca4b0597ba8880" --0000000000008fca4b0597ba8880 Content-Type: text/plain; charset="UTF-8" Hi all, Here is the second proposal for improving SINGA after the code quality proposal [1]. This discussion is about adding code coverage to Travis CI [2]. To enable code coverage, I think we need to fix some stuff first: 1. We need to make the Travis build fails if the test cases fail. Currently, the test cases fail (as can be seen in the log, for example [3]) but the Travis build does not show this problem and finish with "Done. Your build exited with 0." [4]. 2. Some unit tests fail because they run on the wrong architecture, not because there is a problem in the code. So we need to improve the test design. For example, in [5] there is an error "AssertionError: SINGA has not been compiled with CUDA enabled.". I think it is better to run CUDA tests only if the code is compiled for CUDA. This should be the same for all other compilation options and their corresponding test cases. 3. The python tests should run in python machines in Travis not in C++ machines [6]. Currently, the test cases uses C++ machines and install python (with conda) on them to run the python tests, instead of running python machines. 4. The python test for 3.6 [7] and 3.7 [8] should be done on different machines in Travis where the native python packages of the operating system (e.g. ubuntu or Mac) can be used. Currently, both tests happen on one C++ machine and conda is used instead of the python packages of the operating system. This only tests SINGA build with conda, but not with python alone. 5. We need to run the C++ tests [9]. Currently only the python tests are executed in Travis. After these five issues are fixed, we can proceed to 6. Add C++ coverage 7. Add python coverage 8. Show the coverage results 9. Improve the coverage (i.e. add more C++ and python unit tests to cover the code which is not tested) If everyone agrees, I will proceed by creating sub tasks for these issues under SINGA-485. Then we can implement them one by one. Thanks, Moaz [1] http://mail-archives.apache.org/mod_mbox/singa-dev/201909.mbox/%3CCAH%3DWUUbgvL62dZOCRJeEDA-a-JWNEyGvvK1Dn3ZL1YDvLXWktQ%40mail.gmail.com%3E [2] https://issues.apache.org/jira/projects/SINGA/issues/SINGA-485 [3] https://travis-ci.org/apache/singa/jobs/613933817#L3702 [4] https://travis-ci.org/apache/singa/jobs/613933817#L3849 [5] https://travis-ci.org/apache/singa/jobs/613933817#L2879 [6] https://github.com/apache/singa/blob/master/.travis.yml#L19 [7] https://github.com/apache/singa/blob/master/tool/travis/build.sh#L33 [8] https://github.com/apache/singa/blob/master/tool/travis/build.sh#L34 [9] https://github.com/apache/singa/tree/master/test/singa --0000000000008fca4b0597ba8880--