ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: Apache Ignite TeamCity Bot - muted tests
Date Tue, 25 Dec 2018 11:50:00 GMT
Oleg,

Thanks for clarification.

On Tue, Dec 25, 2018 at 2:24 PM oignatenko <oignatenko@gridgain.com> wrote:

> Hi Andrey,
>
> As far as I know tests annotated by @Ignore can be conveniently tracked on
> Teamcity but this requires such tests to be included into JUnit 4 test
> suite
> (org.junit.runners.Suite).
>
> Above requirement currently isn't met by many Ignite tests which are
> included in JUnit 3 suites (junit.framework.TestSuite). This means that
> test
> cases annotated @Ignore simply are hidden when present in such
> old-fashioned
> suites.
>
> We are currently working on migrating Junit 3 suites to JUnit 4. If you are
> interested, work breakdown and progress can be found in JIRA:
> - https://issues.apache.org/jira/browse/IGNITE-10762
>
> -----
>
> With regards to Assume API, it seems more complicated. I have recently read
> that tests that are conditionally skipped using this API are reported by
> JUnit 4 test framework as passed. This is not very convenient.
>
> I think I have seen somewhere mentioned that proper reporting of
> conditionally skipped tests can be achieved only with JUnit 5. But I didn't
> dig deeper there because we haven't yet decided whether it is worth it for
> Ignite to move to JUnit 5
> (https://issues.apache.org/jira/browse/IGNITE-10180).
>
> Given above, I believe that the proper way to mute tests is currently with
> @Ignore and Assume API. It's not 100% ideal but overall looks better than
> the old one.
>
> regards, Oleg
>
>
> Andrew Mashenkov wrote
> > Dmitry,
> >
> > As far as I understand, @IgniteIgnore and fail("") works in similar way
> > and
> > requires test to be muted on TC manually.
> > And we have to unmute these tests on TC right after fix will be merged to
> > master, isn't it?
> >
> > @Ignore and Assume.that works different way comparing to previous two.
> > Junit just exclude such tests from run.
> > So, there is no need to mute\unmute them on TC at all, but...
> > Does TC treat such tests as muted ones and we can see them on "muted
> > tests"
> > page?
> > This may be useful to track tests state if someone forget to remove
> > @ignore
> > while fixing related ticket.
> >
> >
> > On Mon, Dec 17, 2018 at 5:54 PM Anton Vinogradov &lt;
>
> > av@
>
> > &gt; wrote:
> >
> >> >>  In this case, we will not need TeamCity mutes anymore.
> >> Sounds great!
> >>
> >> On Mon, Dec 17, 2018 at 5:23 PM Dmitrii Ryabov &lt;
>
> > somefireone@
>
> > &gt;
> >> wrote:
> >>
> >> > Dmitriy,
> >> >
> >> > with ignore annotations we will need to check forgotten tests
> >> > manually. Thats bad, but we will save some machine time in common
> >> > runs.
> >> > пн, 17 дек. 2018 г. в 17:06, Dmitriy Pavlov &lt;
>
> > dpavlov@
>
> > &gt;:
> >> > >
> >> > > Investigation parameters: Can leave as empty
> >> > > Unmute: Manual (don't use auto because of flaky tests)
> >> > > Text: https://issues.apache.org/jira/browse/IGNITE-NNNNN
> >> > >
> >> > > But anyway, I hope in the nearest future best way to mute test will
> >> be
> >> to
> >> > > ignore it using:
> >> > > @IgniteIgnore
> >> > > or @Ignore
> >> > > or Assume.that()
> >> > >
> >> > > In this case, we will not need TeamCity mutes anymore.
> >> > >
> >> > > I'm not sure which way of ignoring will be best.
> >> > >
> >> > > пн, 17 дек. 2018 г. в 17:02, Anton Vinogradov &lt;
>
> > av@
>
> > &gt;:
> >> > >
> >> > > > Dmitrii,
> >> > > >
> >> > > > My question was mostly about how to mute test properly.
> >> > > >
> >> > > >
> >> > > > On Mon, Dec 17, 2018 at 4:27 PM Dmitrii Ryabov <
> >>
>
> > somefireone@
>
> >>
> >> > > > wrote:
> >> > > >
> >> > > > > Anton,
> >> > > > >
> >> > > > > In the "Mute" column you can find link to the test on TeamCity.
> >> On
> >> > > > > this page you can see test details like run history and
current
> >> > mutes.
> >> > > > > Here you should be able to unmute existing mutes.
> >> > > > > пн, 17 дек. 2018 г. в 16:21, Anton Vinogradov &lt;
>
> > av@
>
> > &gt;:
> >> > > > > >
> >> > > > > > Great feature!
> >> > > > > >
> >> > > > > > Could you please explain how to have a deal with it?
> >> > > > > > How should I mite/unmute tests for now?
> >> > > > > >
> >> > > > > > On Fri, Dec 14, 2018 at 6:50 PM Dmitriy Pavlov <
> >>
>
> > dpavlov@
>
> >> > >
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > We have IgniteIgnore, so we can use it. It seems
behavior is
> >> the
> >> > > > same.
> >> > > > > We
> >> > > > > > > need just to understand in which case we can get
issue ID
> >> from
> >> TC
> >> > > > REST
> >> > > > > (for
> >> > > > > > > the Bot).
> >> > > > > > >
> >> > > > > > > пт, 14 дек. 2018 г. в 17:51, Vyacheslav
Daradur <
> >> >
>
> > daradurvs@
>
> >> > > > >:
> >> > > > > > >
> >> > > > > > > > Dmitry Pavlov,
> >> > > > > > > >
> >> > > > > > > > As far as I know, we are able to use following
> construction
> >> to
> >> > > > ignore
> >> > > > > > > test:
> >> > > > > > > > Assume.assumeTrue("link-to-issue", false);
> >> > > > > > > >
> >> > > > > > > > On Fri, Dec 14, 2018 at 5:44 PM Dmitriy Pavlov
<
> >> >
>
> > dpavlov@
>
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > Thank you, Dmitrii,
> >> > > > > > > > >
> >> > > > > > > > >  I've started unmuting tests starting
from yesterday.
> >> > > > > > > > >
> >> > > > > > > > > Now muted test count was decreased from
3540 to 3408
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&tab=mutedProblems&branch_IgniteTests24Java8=%3Cdefault%3E
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Probably we could automate this activity
by auto-unmute,
> >> but
> >> > > > > hopefully,
> >> > > > > > > > > JUnit4 & @Ignore opportunity will
come earlier.
> >> > > > > > > > >
> >> > > > > > > > > Sincerely,
> >> > > > > > > > > Dmitriy Pavlov
> >> > > > > > > > >
> >> > > > > > > > > пт, 14 дек. 2018 г. в 17:24,
Dmitrii Ryabov <
> >> > > >
>
> > somefireone@
>
> >> > > > > >:
> >> > > > > > > > >
> >> > > > > > > > > > Hello, Igniters!
> >> > > > > > > > > >
> >> > > > > > > > > > Our Bot got new functionality -
it shows tests muted
> on
> >> > > > TeamCity
> >> > > > > and
> >> > > > > > > > their
> >> > > > > > > > > > ticket status [1].
> >> > > > > > > > > >
> >> > > > > > > > > > So, we can see forgotten mutes
for resolved tickets
> >> now.
> >> > > > > > > > > >
> >> > > > > > > > > > Igniters with rights to unmute
tests on TC - please,
> >> check
> >> > this
> >> > > > > page
> >> > > > > > > > > > periodically for forgotten mutes.
> >> > > > > > > > > >
> >> > > > > > > > > > [1] https://mtcga.gridgain.com/mutes.html
> >> > > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Best Regards, Vyacheslav D.
> >> > > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


-- 
Best regards,
Andrey V. Mashenkov

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message