Mailing-List: contact dev-help@poi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "POI Developers List" <dev@poi.apache.org>
MIME-Version: 1.0
In-Reply-To: 
 <CAM+TppK2pC3x5xkj+N_vs8Bkd5K-brQt3Ht9CXkLxZEN0GN=ag@mail.gmail.com>
References: 
 <CAM+TppKFjTM4Jo4JcEjH+O+v9uqtg_i4kSHXnzAA-KYWyw_1KA@mail.gmail.com>
 <CABdJj56PdcqTZRqmo9Ltk36FpSN_LxRLj5yHXT7BXCkfL8YbSA@mail.gmail.com>
 <CAM+TppK2pC3x5xkj+N_vs8Bkd5K-brQt3Ht9CXkLxZEN0GN=ag@mail.gmail.com>
From: "Javen O'Neal" <javenoneal@gmail.com>
Date: Sun, 3 Jan 2016 01:30:13 -0800
Message-ID: 
 <CAM+TppLSK9=+P-B8YGve8Ycn6Lw3p=K4Dby-EO+gtuU67Y3s3w@mail.gmail.com>
Subject: Re: Detecting when ignored unit tests pass
To: POI Developers List <dev@poi.apache.org>
Content-Type: text/plain; charset=UTF-8

Here's a working example of what I was talking about. This will break
the build as soon as the bug is patched (prompting a unit test
update). Because this unit test is run, any side-effects it has also
occur, which differs from not running the unit test altogether. In
this case, the side effect is modifying
/test-data/slideshow/SampleShow.pptx.

https://svn.apache.org/viewvc/poi/trunk/src/ooxml/testcases/org/apache/poi/xslf/usermodel/TestXSLFSlideShowFactory.java?r1=1722707&r2=1722706&pathrev=1722707

public final class TestXSLFSlideShowFactory extends BaseTestSlideShowFactory {
    private static final String removeExpectedExceptionMsg =
            "This functionality this unit test is trying to test is
now passing. " +
            "The unit test needs to be updated by deleting the
expected exception code. Status and close any related bugs.";

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void testFactoryFromFile() throws Exception {
        // Remove thrown.* when bug 58779 is resolved
        thrown.expect(AssertionError.class);
        thrown.expectMessage("SampleShow.pptx sample file was modified
as a result of closing the slideshow");
        thrown.reportMissingExceptionWithMessage("Bug 58779: " +
removeExpectedExceptionMsg);

        testFactoryFromFile(filename);
    }
}

On Sat, Jan 2, 2016 at 5:11 PM, Javen O'Neal <javenoneal@gmail.com> wrote:
> On Sat, Jan 2, 2016 at 2:54 PM, Dominik Stadler <dominik.stadler@gmx.at> wrote:
>> Adding a test-suite which looks at bugzilla will cause additional
>> web-accesses during CI runs and might cause unstability if bugzilla is
>> down/unreachable/flaky or changes web-address (just happened a year
>> ago!)... I saw this being used locally at my company but am not sure if it
>> is a good idea for an open source project where code is built all over the
>> place.
>
> I was thinking of mentioning the bug number in the error message, but
> assume the user will figure out what the bug URL is and change the
> status as needed. No need to integrate bugzilla with the test suite.
> Unit tests should be able to be run without an internet connection. If
> the test is named something like "test12345" or "bug12345", someone
> might be able to figure out what this is about--but we need to be
> crystal clear that if the test breaks, it's because the broken feature
> now works as expected, not because a working feature broke, and to fix
> the unit test rather than the Java code.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org