Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
From: Apekshit Sharma <appy@cloudera.com>
Date: Mon, 16 May 2016 16:38:35 -0700
Message-ID: <CAAjhxrqa3BcOaME3bTmgnyHOw1tDSAGzH=rV7JNyc_1dZDH_Kg@mail.gmail.com>
Subject: Smart Flaky Handler
To: dev@hbase.apache.org
Content-Type: multipart/related; boundary=089e013d0c304033e30532fe2149
archived-at: Mon, 16 May 2016 23:39:18 -0000

--089e013d0c304033e30532fe2149
Content-Type: multipart/alternative; boundary=089e013d0c304033de0532fe2148

--089e013d0c304033de0532fe2148
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

 This mail is to introduce the work to tackle the flaky tests in our build.

*Why is it important?*
- Our build history sucks, last 175 post-commit runs failed. We need to
make it useful.
- To better understand our code=E2=80=99s testing status, more importantly =
it=E2=80=99s
weak points.
- We know those 2-3 tests which keep failing every now and then, but not
those ~10 nasty ones which fail like 1 out of 50 times, and screw our build=
.
- This isn=E2=80=99t something that can be done manually on a daily basis. =
We need
automation.

*Changes made so far:*
Code changes: HBASE-15839
<https://issues.apache.org/jira/browse/HBASE-15839>  (Umbrella issue)

*Jenkins changes:*


[Diagram link:
https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202=
016-05-16%20at%204.02.46%20PM.png
]
=E2=80=8B
*(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds of
post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to
find flaky tests. Frequency of run determines how fast we catch test
regressions. So if we run it every 4 hours, any test which started failing
in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.

*(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The aim
is to run this job back-to-back to collect as many runs as we can. Higher
the run rate, the better will be our system at catching the flaky tests. We
currently run it hourly. so we=E2=80=99ll be able to keep track of flaky te=
sts with
~5% failure rate or more.

*Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky tests=
.


*So what if a bad commit makes a good test bad?*
Since the test is not bad, it=E2=80=99ll run in next post-commit and will f=
ail.
Next run of HBase-Find-Flaky-Tests will  pick it up and blacklist it.
Blacklisting will help keep the post-commit job and more importantly
pre-commit job clean, a problem we face quite often.

*Are we just tucking away are shit?*
Nope, this will help us:
- first, Maintain a list of bad test (we lack that today).
- second, make our build greener to the point that a failed/red build is
something we worry about seriously.

Once we are confident that the system is working fine, we=E2=80=99ll setup =
up
HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs know
about the bad tests. If it remains hidden somewhere in a jenkins job=E2=80=
=99s
archive, it=E2=80=99s unlike that we=E2=80=99ll actively work on getting th=
em fixed :).

I'll keep posting further updates on this thread.

-- Appy

--089e013d0c304033de0532fe2148
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">

This mail is to introduce the work to tackle the flaky tests in our build.<=
br><br><b>Why is it important?</b><br>- Our build history sucks, last 175 p=
ost-commit runs failed. We need to make it useful.<div>- To better understa=
nd our code=E2=80=99s testing status, more importantly it=E2=80=99s weak po=
ints.<br>- We know those 2-3 tests which keep failing every now and then, b=
ut not those ~10 nasty ones which fail like 1 out of 50 times, and screw ou=
r build.<br>- This isn=E2=80=99t something that can be done manually on a d=
aily basis. We need automation.<br><br><b>Changes made so far:</b><br>Code =
changes: <a href=3D"https://issues.apache.org/jira/browse/HBASE-15839" targ=
et=3D"_blank">HBASE-15839</a> =C2=A0(Umbrella issue)<br><br></div><div><b>J=
enkins changes:</b></div><div><br></div><img src=3D"cid:ii_ioan2ijz0_154bbe=
3a016cfacc" width=3D"542" height=3D"290"><br><div>[Diagram link:=C2=A0<a hr=
ef=3D"https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Sh=
ot%202016-05-16%20at%204.02.46%20PM.png" target=3D"_blank">https://issues.a=
pache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%2=
04.02.46%20PM.png</a>]</div><div>=E2=80=8B<br></div><div><u>(new job) HBase=
-Find-Flaky-Tests</u>: Gets test reports of recent builds of post-commit jo=
b (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to find flaky tests.=
 Frequency of run determines how fast we catch test regressions. So if we r=
un it every 4 hours, any test which started failing in post-commit job (TRU=
NK_matrix) in last 4 hour will be blacklisted.<br><br></div><div><u>(new jo=
b) HBase-Flaky-Tests</u>: This job runs only the flaky tests. The aim is to=
 run this job back-to-back to collect as many runs as we can. Higher the ru=
n rate, the better will be our system at catching the flaky tests. We curre=
ntly run it hourly. so we=E2=80=99ll be able to keep track of flaky tests w=
ith ~5% failure rate or more.<br><br></div><div><u>Post-commit (TRUNK_matri=
x) and pre-commit jobs</u>: Exclude these flaky tests.<br><br><br><b>So wha=
t if a bad commit makes a good test bad?</b><br>Since the test is not bad, =
it=E2=80=99ll run in next post-commit and will fail. Next run of HBase-Find=
-Flaky-Tests will =C2=A0pick it up and blacklist it. Blacklisting will help=
 keep the post-commit job and more importantly pre-commit job clean, a prob=
lem we face quite often.<br><br><b>Are we just tucking away are shit?</b><b=
r>Nope, this will help us:<br>- first, Maintain a list of bad test (we lack=
 that today).<br>- second, make our build greener to the point that a faile=
d/red build is something we worry about seriously.<br><br>Once we are confi=
dent that the system is working fine, we=E2=80=99ll setup up HBase-Find-Fla=
ky-Tests job to send reports to dev@hbase so that devs know about the bad t=
ests. If it remains hidden somewhere in a jenkins job=E2=80=99s archive, it=
=E2=80=99s unlike that we=E2=80=99ll actively work on getting them fixed :)=
.=C2=A0</div><div><br></div><div>I&#39;ll keep posting further updates on t=
his thread.</div><div><br><div>-- Appy</div></div></div>

--089e013d0c304033de0532fe2148--

--089e013d0c304033e30532fe2149--