Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Mon, 20 Feb 2017 08:05:44 +0000 (UTC)
From: "Stas Levin (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.13040491.1486323536000.8735.1487577944212@Atlassian.JIRA>
In-Reply-To: <JIRA.13040491.1486323536000@Atlassian.JIRA>
References: <JIRA.13040491.1486323536000@Atlassian.JIRA> <JIRA.13040491.1486323536473@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (BEAM-1399) Code coverage numbers are not
 accurate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 20 Feb 2017 08:06:05 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15873590#comment-15873590 ] 

Stas Levin edited comment on BEAM-1399 at 2/20/17 8:04 AM:
-----------------------------------------------------------

Thanks for the tip [~dhalperi@google.com], I wasn't aware the {{beam-sdks-java-javadoc}} module depended on all the other ones.

From preliminary experiments, looks like {{report-aggregate}} aggregates reports across dependent modules, so that the resulting report is as if you had executed the regular {{report}} and merged them all together (visually). 
In terms of coverage, I'm still seeing the 69% for {{beam-sdks-java-core}} both when using {{report}} specifically for {{beam-sdks-java-core}} and {{report-aggregate}} for the {{beam-sdks-java-javadoc}} module. This may indicate that {{report-aggregate}} does not capture the additional {{@RunnableOnService/@NeedsRunner}} tests we are after, at least not out-of-the box. 
In addition, test class coverage is not reported (neither by {{report}} nor by {{report-aggregate}}), only production code ^2^.

Essentially we have the following major issues:
# Record coverage across modules 
#* Might be possible using {{maven-antrun-plugin}} and/or copying {{.class}} / {{.java}} files of dependent modules to {{beam-sdks-java-javadoc}}, as described in ^1^.
# Report test class coverage 
#* Might be possible using {{maven-antrun-plugin}} since it gives fine-grained control over what's reported.
# Merge coverage provided by different executions of the same test class 
#* As I dive deeper into this, I'm inclined to think that the {{jacoco-maven-plugin}} does not support merging coverage for a class executed from different modules due to ^3^, which brings me back to separating {{@RunnableOnService/@NeedsRunner}} into their own test classes and the rest of the goodies behind curtain number (2) detailed in my first comment above.


1. https://groups.google.com/forum/#!searchin/jacoco/integration$20tests|sort:relevance/jacoco/_yuHGr_Kp6s/irM031B6KNAJ
2. http://stackoverflow.com/questions/34483904/jacoco-maven-plugin-include-test-sources
3.  http://www.eclemma.org/jacoco/trunk/doc/classids.html


was (Author: staslev):
Thanks for the tip [~dhalperi@google.com], I wasn't aware the {{beam-sdks-java-javadoc}} module depended on all the other ones.

From preliminary experiments, looks like {{report-aggregate}} aggregates reports across dependent modules, so that the resulting report is as if you had executed the regular {{report}} and merged them all together (visually). 
In terms of coverage, I'm still seeing the 69% for {{beam-sdks-java-core}} both when using {{report}} specifically for {{beam-sdks-java-core}} and {{report-aggregate}} for the {{beam-sdks-java-javadoc}} module. This may indicate that {{report-aggregate}} does not capture the additional {{@RunnableOnService/@NeedsRunner}} tests we are after, at least not out-of-the box. 
In addition, test class coverage is not reported (neither by {{report}} nor by {{report-aggregate}}), only production code ^2^.

Essentially we have the following major issues:
# Record coverage across modules 
#* Might be possible using {{maven-antrun-plugin}} and/or copying {{.class}} / {{.java}} files of dependent modules to {{beam-sdks-java-javadoc}}, as described in ^1^.
# Report test class coverage 
#* Might be possible using {{maven-antrun-plugin}} since it gives fine-grained control over what's reported.
# Merge coverage provided by different executions of the same test class 
#* As I dive deeper into this, I'm inclined to think that the {{jacoco-maven-plugin}} does not support merging coverage for a test class executed from different modules due to ^3^, which brings me back to separating {{@RunnableOnService/@NeedsRunner}} into their own test classes and the rest of the goodies behind curtain number (2) detailed in my first comment above.


1. https://groups.google.com/forum/#!searchin/jacoco/integration$20tests|sort:relevance/jacoco/_yuHGr_Kp6s/irM031B6KNAJ
2. http://stackoverflow.com/questions/34483904/jacoco-maven-plugin-include-test-sources
3.  http://www.eclemma.org/jacoco/trunk/doc/classids.html

> Code coverage numbers are not accurate
> --------------------------------------
>
>                 Key: BEAM-1399
>                 URL: https://issues.apache.org/jira/browse/BEAM-1399
>             Project: Beam
>          Issue Type: Bug
>          Components: build-system, sdk-java-core, testing
>            Reporter: Daniel Halperin
>              Labels: newbie, starter
>
> We've started adding Java Code Coverage numbers to PRs using the jacoco plugin. However, we are getting very low coverage reported for things like the Java SDK core.
> My belief is that this is happening because we test the bulk of the SDK not in the SDK module , but in fact in the DirectRunner and other similar modules.
> JaCoCo has a {{report:aggregate}} target that might do the trick, but with a few minutes of playing with it I wasn't able to make it work satisfactorily. Basic work in https://github.com/apache/beam/pull/1800
> This is a good "random improvement" issue for anyone to pick up.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)