ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitrii Ryabov <somefire...@gmail.com>
Subject Re: Workflow improvement
Date Thu, 13 Sep 2018 13:05:20 GMT
Hi, Dmitriy,

Can you give me rights to stop my builds on TeamCity? Working on the TCH, I
run a lot of builds, and I don't like to ask other people to stop builds
too often.

2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <somefireone@gmail.com>:

> Hi, Dmitriy,
>
> I made PRs in my fork for test purposes. Real PRs were made to the GitHub
> mirror, and one of them is already merged by D. Govorukhin. PR with GitHub
> statuses [1] is ready for review. PR with JIRA comment will be ready in a
> few days.
>
> [1] https://github.com/apache/ignite-teamcity-bot/pull/5
>
> 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com>:
>
>> Hi Dmitrii,
>>
>> I deployed change with blockers summary of failures at the top of PR
>> result
>> page. The Bot is migrating entries now, I hope it will be done in 1-4
>> hours.
>>
>> I noticed your PRs are created in your own fork, not in
>> https://github.com/apache/ignite-teamcity-bot
>> Could you please create unmerged PR in GitHub mirror so it could be merged
>> after reviewing by the apply-pr script.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <somefireone@gmail.com>:
>>
>> > Hi, Dmitriy,
>> >
>> > I created a table with possible blockers [1] for the page with the
>> latest
>> > build analysis. Anton K. has reviewed it. Can you check and merge it?
>> >
>> > Also, I created a button to notify PR about analisis results [2]. I used
>> > GitHub statuses (example [3]), but to work it needs a token with push
>> > permission. As Apache PMC, can you acquire such token? I think statuses
>> are
>> > much more notable than comments.
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-9376
>> > [2] https://issues.apache.org/jira/browse/IGNITE-9377
>> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
>> >
>> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com>:
>> >
>> > > Not sure I understood what bot statistic reduction means.
>> > >
>> > > As far as I understood, you also mean locating failure automatically
>> with
>> > > re-runs on specific git revisions (hashes). I found it will be a good
>> > > contribution to TC Bot for both flaky and non-flaky failures
>> detection.
>> > E.g
>> > > failure occurred after 10 commits merged to master. How to find out
>> bad
>> > > commit? Automatic location (analog of bisecting, but using TC test)
>> will
>> > be
>> > > really helpful. But it is not a simple contribution.
>> > >
>> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <somefireone@gmail.com>:
>> > >
>> > > > Hi, Dmitriy,
>> > > >
>> > > > Good, I'll create tickets for this steps.
>> > > >
>> > > > Stable suites are a good idea, but also we need to do some actions
>> > when a
>> > > > flaky test will appear in stable suite of master branch run. To find
>> > out
>> > > a
>> > > > guilty commit, we should run previous commits on empty TeamCity's
>> > queue.
>> > > > This runs will reduce bot's statistics for the latest commits.
>> > > >
>> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com>:
>> > > >
>> > > > > Hi Dmitriy,
>> > > > >
>> > > > > Sounds like a plan ;) I totally agree.
>> > > > > Just one proposal: I would like to avoid hiding any test
>> failures. 2
>> > > > > separate tables named 'Possible Blockers' and 'Other failures'
>> should
>> > > be
>> > > > > completely clear. Comment to PR can contain only first category.
>> > > > >
>> > > > > We had a private discussion with Anton K. and he proposed a very
>> > > > > interesting thing, I would like to bring it here. We can add
>> > > > configuration
>> > > > > into TC bot and mark some tests and some suites as 'Known Stable'
>> It
>> > > > means
>> > > > > that suite or test failure should be considered as a blocker
for
>> > merge
>> > > > > every time it fails, even if fail rate is non-zero. Very first
>> list
>> > of
>> > > > such
>> > > > > suites are
>> > > > >  - Build Apache Ignite
>> > > > >  - License check
>> > > > > And tests:
>> > > > >  - .Net API Parity check
>> > > > > Please share your vision.
>> > > > >
>> > > > > Meanwhile, I've updated the TC bot.
>> > > > > - Underlying Apache Ignite DB was refactored to use lower
>> partitions
>> > > > count,
>> > > > > so restart should be faster.
>> > > > > - Now 100 builds are saved as our failure rate statistics basis.
>> > > > Flakiness
>> > > > > border was not changed, so more test now will be considered as
>> flaky.
>> > > > > - Now 4 builds required to be failed or timed out in a row before
>> > > > > notification is sent.
>> > > > >
>> > > > > Sincerely,
>> > > > > Dmitriy Pavlov
>> > > > >
>> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <
>> somefireone@gmail.com>:
>> > > > >
>> > > > > > Hi, Dmitriy,
>> > > > > >
>> > > > > > I propose the next steps:
>> > > > > >
>> > > > > > 1. Show current 0% tests in a separate table at the top
of the
>> > > analysis
>> > > > > > results page. Thus, we'll see most suspicious (new or very
>> flaky)
>> > > tests
>> > > > > > firstly. Or we can hide other tests under "More >>"
button, like
>> > top
>> > > > long
>> > > > > > running tests.
>> > > > > > 2. Create a button by clicking on which the info about 0%
tests
>> > will
>> > > be
>> > > > > > written in the PR.
>> > > > > > 3. Replace button by webhook for TeamCity (for Run All),
which
>> will
>> > > > start
>> > > > > > analysis on TCH and write results in the PR.
>> > > > > > 4. ...
>> > > > > >
>> > > > > > What do you think?
>> > > > > >
>> > > > > >
>> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <somefireone@gmail.com
>> >:
>> > > > > >
>> > > > > > > I think we should check not N last runs, but all runs
in last
>> N
>> > > days.
>> > > > > > >
>> > > > > > > A simple rule to detect flaky fails by hands - get
test
>> history
>> > > > ordered
>> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we
can get
>> list
>> > > of
>> > > > > > > failures from TC. We don't need to check successfull
runs,
>> > because
>> > > > they
>> > > > > > are
>> > > > > > > uninformative for our needs.
>> > > > > > >
>> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <
>> dpavlov.spb@gmail.com
>> > >:
>> > > > > > >
>> > > > > > >> Hi Dmitriy,
>> > > > > > >>
>> > > > > > >> The Bot is able to detect a frequent change of
test status,
>> but
>> > > > > > currently
>> > > > > > >> only 50 last runs count. Same is true for the failure
rate.
>> > > > > > >>
>> > > > > > >> This value can be easily changed to 70 or 100,
moreover, the
>> > auto
>> > > > > > >> trigger feature gives us much more builds.
>> > > > > > >>
>> > > > > > >> We can improve these rules. We can add not only
status
>> change,
>> > but
>> > > > > > status
>> > > > > > >> change without any code changes. We can somehow
save this
>> data
>> > in
>> > > > > > RunStat
>> > > > > > >> class. Let's create a better rule, and later we
can code it.
>> > > > > > >>
>> > > > > > >> Sincerely,
>> > > > > > >> Dmitriy Pavlov
>> > > > > > >>
>> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov
<
>> > > somefireone@gmail.com
>> > > > >:
>> > > > > > >>
>> > > > > > >> > I think plugin will be more pretty looking,
but comments
>> can
>> > > > contain
>> > > > > > any
>> > > > > > >> > information, so they can be more usefull.
I agree with your
>> > idea
>> > > > to
>> > > > > > >> create
>> > > > > > >> > bot instead of plugin.
>> > > > > > >> >
>> > > > > > >> > As for fail rate - I'm not sure it is working
as you
>> describe.
>> > > > > > >> > I'm looking on my runAll [1]. There is
>> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
>> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%.
But it is flaky
>> and
>> > > > fails
>> > > > > in
>> > > > > > >> > master branch [2].
>> > > > > > >> >
>> > > > > > >> > [1]
>> > > > > > >> >
>> > > > > > >> > https://mtcga.gridgain.com/pr.
>> html?serverId=apache&suiteId=I
>> > > > > > >>
>> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&
>> action=Latest
>> > > > > > >> > [2]
>> > > > > > >> >
>> > > > > > >> > https://ci.ignite.apache.org/p
>> roject.html?projectId=IgniteTe
>> > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
>> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
>> > > > > > >> 24Java8=__all_branches__&itemsCount=50
>> > > > > > >> >
>> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov
<
>> > > dpavlov.spb@gmail.com
>> > > > >:
>> > > > > > >> >
>> > > > > > >> > > Hi Dmitrii,
>> > > > > > >> > >
>> > > > > > >> > > I'm not sure we're able to install Github
apps to Apache
>> > > > mirrors.
>> > > > > > >> > >
>> > > > > > >> > > The simplest solution, what can be as
efficient as a
>> plugin,
>> > > is
>> > > > > fake
>> > > > > > >> > MTCGA
>> > > > > > >> > > bot account in Github, which will provide
PR comments
>> using
>> > > > Github
>> > > > > > >> > program
>> > > > > > >> > > interface. What do you think?
>> > > > > > >> > >
>> > > > > > >> > > A new test failure can be identified
by the Ignite TC
>> Bot by
>> > > > > master
>> > > > > > >> > recent
>> > > > > > >> > > fail rate = 0.0%. The same rule can be
applied to timed
>> out
>> > > > > suites.
>> > > > > > >> > >
>> > > > > > >> > > Sincerely,
>> > > > > > >> > > Dmitriy Pavlov
>> > > > > > >> > >
>> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii
Ryabov <
>> > > > > somefireone@gmail.com
>> > > > > > >:
>> > > > > > >> > >
>> > > > > > >> > > > Hello, Igniters!
>> > > > > > >> > > >
>> > > > > > >> > > > I want to suggest improvement for
TeamCity Helper [1]
>> – we
>> > > > need
>> > > > > an
>> > > > > > >> easy
>> > > > > > >> > > way
>> > > > > > >> > > > to get list of failed tests that
don’t fall in the
>> master
>> > > > > branch.
>> > > > > > >> These
>> > > > > > >> > > > tests should:
>> > > > > > >> > > > * fail in the PR
>> > > > > > >> > > > * not fail in the master
>> > > > > > >> > > > * not be flaky.
>> > > > > > >> > > >
>> > > > > > >> > > > Also, I want to suggest to create
a GitHub plugin,
>> which
>> > > will
>> > > > > > >> notify PR
>> > > > > > >> > > if
>> > > > > > >> > > > it has such tests. PR will have
a marker, which
>> > > > allows/prohibits
>> > > > > > >> merge.
>> > > > > > >> > > > This marker will be shown near PR
conflicts.
>> > > > > > >> > > >
>> > > > > > >> > > > Allowing marker will be shown in
case:
>> > > > > > >> > > > * no new fails.
>> > > > > > >> > > >
>> > > > > > >> > > > Prohibiting marker will be shown
in cases:
>> > > > > > >> > > > * new fails – tests must be fixed.
>> > > > > > >> > > > * new timed out test suite – suite
should be restarted
>> or
>> > > > tests
>> > > > > > >> must be
>> > > > > > >> > > > fixed.
>> > > > > > >> > > > * runAll wasn’t launched – tests
must be launched.
>> > > > > > >> > > >
>> > > > > > >> > > > This will make test checks much
faster and easier.
>> Also,
>> > > this
>> > > > > will
>> > > > > > >> > > decrease
>> > > > > > >> > > > the number of merges with new failed
tests made by
>> > > inattention
>> > > > > to
>> > > > > > >> the
>> > > > > > >> > > > tests.
>> > > > > > >> > > >
>> > > > > > >> > > > Further, we can expand the plugin
by adding new checks,
>> > > > showing
>> > > > > PR
>> > > > > > >> > > quality.
>> > > > > > >> > > >
>> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message