hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)
Date Fri, 30 Oct 2015 16:18:42 GMT
Hi Folks,

Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS]
reducing our load on builds.a.o' on reasoning or if you have concerns.

Our project job view now has a brief summary of what's going on:

https://builds.apache.org/view/H-L/view/HBase/

* Where we had matrix builds, I confirmed they're doing the same thing
(in at least one test-configuration) as the non-matrix version
* I disabled jobs that were duplicating work done in matrix jobs, or
that only worked on a tag
* Made sure all non-disabled 0.98+ jobs are set to notify jira +
builds mailing list
* I updated pre-1.1 jobs to look for changes 1/day
* I updated 1.1 jobs to look for changes every 8 hours
* I updated 1.2, 1.3, and trunk to look for changes every 4 hours

There's still some non-pressing clean up work to do, so I'll file a
jira later today just to make sure it's all recorded somewhere.

On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <busbey@cloudera.com> wrote:
> Added new HBase-1.3 build or branch-1 and changed the existing 1.2 builds to
> use branch-1.2. (ref HBASE-13912)
>
> On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <busbey@cloudera.com> wrote:
>>
>> I've add a new build test to run the ITs under sunny day conditions using
>> minicluster for both java 7 and java 8 on the 1.2 line.
>>
>> https://builds.apache.org/job/HBase-1.2-IT/
>>
>> I haven't turned on notifications yet, because BulkIngest and TestIngest
>> are read. details on HBASE-13750
>>
>> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <busbey@cloudera.com> wrote:
>>>
>>> sigh. that should have been "is now a matrix build".
>>>
>>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <busbey@cloudera.com> wrote:
>>>>
>>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8 in
>>>> parallel.
>>>>
>>>> I saved the old job for now and left it disabled[2].
>>>>
>>>>
>>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/
>>>> [2]:
>>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/
>>>>
>>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <busbey@cloudera.com>
>>>> wrote:
>>>>>
>>>>> Sorry for the noise. I also updated the checkstyle step on HBase-TRUNK
>>>>>
>>>>> from
>>>>>     mvn checkstyle:checkstyle
>>>>> to
>>>>>     mvn -DskipTests package checkstyle:checkstyle
>>>>>
>>>>> to deal with the same issue. Looks all clear now.
>>>>>
>>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <busbey@cloudera.com>
>>>>> wrote:
>>>>>>
>>>>>> Updated the following builds:
>>>>>>
>>>>>> * HBase-TRUNK
>>>>>>
>>>>>> moved from
>>>>>>
>>>>>>     mvn clean compile findbugs:findbugs
>>>>>>
>>>>>> to
>>>>>>
>>>>>>    mvn clean -DskipTests package findbugs:findbugs
>>>>>>
>>>>>> To work around the known issue where we can't do a bootstrap compile
>>>>>> without previous install or remote SNAPSHOT artifacts. (recently
triggered
>>>>>> by the addition of the procedure module on master)
>>>>>>
>>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino219@gmail.com>
wrote:
>>>>>>>
>>>>>>> Make findbugs and checkstyle plugins always run.
>>>>>>> The default behavior only publishes result for stable builds,
but
>>>>>>> master is
>>>>>>> not always stable and sometimes we introduce new warnings in
unstable
>>>>>>> builds(see builds 6310-6314).
>>>>>>> And findbugs and checkstyle will not fail unless we abort the
>>>>>>> building
>>>>>>> process I think, so it is safe to turn it on every time.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>>>>>>>
>>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report.
>>>>>>> >
>>>>>>> > Cheers
>>>>>>> >
>>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino219@gmail.com>
wrote:
>>>>>>> >
>>>>>>> > > Going to change the config of HBase-TRUNK jenkins to
show
>>>>>>> > > findbugs,
>>>>>>> > > checkstyle and jacoco coverage report.
>>>>>>> > > The config has been tested on
>>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for
nearly 30
>>>>>>> > > times.
>>>>>>> > Not
>>>>>>> > > much different from HBase-TRUNK unless it runs ~30%
slower(the
>>>>>>> > > overhead
>>>>>>> > of
>>>>>>> > > collecting information for code coverage).
>>>>>>> > > Thanks.
>>>>>>> > >
>>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <apurtell@apache.org>:
>>>>>>> > >
>>>>>>> > > > +1, thanks a lot for improving our build hygiene.
>>>>>>> > > >
>>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar
>>>>>>> > > > <enis.soz@gmail.com>
>>>>>>> > > wrote:
>>>>>>> > > >
>>>>>>> > > > > Thanks Sean. This is great.
>>>>>>> > > > >
>>>>>>> > > > > Enis
>>>>>>> > > > >
>>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey
>>>>>>> > > > > <busbey@cloudera.com>
>>>>>>> > > > wrote:
>>>>>>> > > > >
>>>>>>> > > > > > FYI I just finished chasing down the
breakage for mvn site
>>>>>>> > > > > > on all
>>>>>>> > > patch
>>>>>>> > > > > > builds.
>>>>>>> > > > > >
>>>>>>> > > > > > HBASE-13191 consolidates the few places
in the test-patch
>>>>>>> > > > > > code
>>>>>>> > where
>>>>>>> > > we
>>>>>>> > > > > > hard coded MAVEN_OPTS.
>>>>>>> > > > > >
>>>>>>> > > > > > If you look at the PreCommit job now,
we use the "set
>>>>>>> > > > > > environment
>>>>>>> > > > > > variables" option to set MAVEN_OPTS
and then everything
>>>>>>> > > > > > else
>>>>>>> > respects
>>>>>>> > > > > that
>>>>>>> > > > > > setting.
>>>>>>> > > > > >
>>>>>>> > > > > > I set the initial value to be a combination
of the memory
>>>>>>> > limitations
>>>>>>> > > > > we've
>>>>>>> > > > > > been actually running with (the ~6G
was getting ignored)
>>>>>>> > > > > > and the
>>>>>>> > > > permgen
>>>>>>> > > > > > needed for site.
>>>>>>> > > > > >
>>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData
-XX:MaxPermSize=256m
>>>>>>> > > > > >
>>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack
<stack@duboce.net>
>>>>>>> > > > > > wrote:
>>>>>>> > > > > >
>>>>>>> > > > > > > I just made TRUNK and branch-1
builds use same jvm as
>>>>>>> > > > > > > patch-build
>>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and
I set the MAVEN_OPTS to
>>>>>>> > > > > > > be the
>>>>>>> > > same
>>>>>>> > > > as
>>>>>>> > > > > > > those of trunk build too, setting
>>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"... it
>>>>>>> > had
>>>>>>> > > > been
>>>>>>> > > > > > > 3000.
>>>>>>> > > > > > >
>>>>>>> > > > > > > Yours,
>>>>>>> > > > > > > St.Ack
>>>>>>> > > > > > >
>>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM,
Stack <stack@duboce.net>
>>>>>>> > > > > > > wrote:
>>>>>>> > > > > > >
>>>>>>> > > > > > > > I upped hadoopqa retention
to keep last 100 builds and
>>>>>>> > > > > > > > or last
>>>>>>> > 7
>>>>>>> > > > > days,
>>>>>>> > > > > > > > whichever comes first.
>>>>>>> > > > > > > > St.Ack
>>>>>>> > > > > > > >
>>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38
AM, Stack
>>>>>>> > > > > > > > <stack@duboce.net>
>>>>>>> > wrote:
>>>>>>> > > > > > > >
>>>>>>> > > > > > > >> Branch-1 and master have
stabilized and now run mostly
>>>>>>> > > > > > > >> blue
>>>>>>> > > (give
>>>>>>> > > > or
>>>>>>> > > > > > > take
>>>>>>> > > > > > > >> the odd failure) [1][2].
Having a mostly blue branch-1
>>>>>>> > > > > > > >> has
>>>>>>> > > helped
>>>>>>> > > > us
>>>>>>> > > > > > > >> identify at least one
destabilizing commit in the last
>>>>>>> > > > > > > >> few
>>>>>>> > days,
>>>>>>> > > > > maybe
>>>>>>> > > > > > > two;
>>>>>>> > > > > > > >> this is as it should be
(smile).
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >> Lets keep our builds blue.
If you commit a patch, make
>>>>>>> > > > > > > >> sure
>>>>>>> > > > > subsequent
>>>>>>> > > > > > > >> builds stay blue. You
can subscribe to
>>>>>>> > builds@hbase.apache.org
>>>>>>> > > to
>>>>>>> > > > > get
>>>>>>> > > > > > > >> notice of failures if
not already subscribed.
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >> Thanks,
>>>>>>> > > > > > > >> St.Ack
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >> 1.
>>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
>>>>>>> > > > > > > >> 2.
>>>>>>> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at
4:41 PM, Stack
>>>>>>> > > > > > > >> <stack@duboce.net>
>>>>>>> > > wrote:
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >>> A few notes on testing.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> Too long to read,
infra is more capable now and after
>>>>>>> > > > > > > >>> some
>>>>>>> > > work,
>>>>>>> > > > we
>>>>>>> > > > > > are
>>>>>>> > > > > > > >>> seeing branch-1 and
trunk mostly running blue. Lets
>>>>>>> > > > > > > >>> try and
>>>>>>> > > keep
>>>>>>> > > > it
>>>>>>> > > > > > > this
>>>>>>> > > > > > > >>> way going forward.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> Apache Infra has new,
more capable hardware.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> A recent spurt of
test fixing combined with more
>>>>>>> > > > > > > >>> capable
>>>>>>> > > hardware
>>>>>>> > > > > > seems
>>>>>>> > > > > > > >>> to have gotten us
to a new place; tests are mostly
>>>>>>> > > > > > > >>> passing
>>>>>>> > now
>>>>>>> > > on
>>>>>>> > > > > > > branch-1
>>>>>>> > > > > > > >>> and master.  Lets
try and keep it this way and start
>>>>>>> > > > > > > >>> to trust
>>>>>>> > > our
>>>>>>> > > > > > test
>>>>>>> > > > > > > runs
>>>>>>> > > > > > > >>> again.  Just a few
flakies remain.  Lets try and nail
>>>>>>> > > > > > > >>> them.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> Our tests now run
in parallel with other test suites
>>>>>>> > > > > > > >>> where
>>>>>>> > > > previous
>>>>>>> > > > > > we
>>>>>>> > > > > > > >>> ran alone. You can
see this sometimes when our zombie
>>>>>>> > detector
>>>>>>> > > > > > reports
>>>>>>> > > > > > > >>> tests from another
project altogether as lingerers
>>>>>>> > > > > > > >>> (To be
>>>>>>> > > fixed).
>>>>>>> > > > > > > Some of
>>>>>>> > > > > > > >>> our tests are failing
because a concurrent hbase run
>>>>>>> > > > > > > >>> is
>>>>>>> > undoing
>>>>>>> > > > > > > classes and
>>>>>>> > > > > > > >>> data from under it.
Also, lets fix.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> Our tests are brittle.
It takes 75minutes for them to
>>>>>>> > complete.
>>>>>>> > > > > Many
>>>>>>> > > > > > > >>> are heavy-duty integration
tests starting up multiple
>>>>>>> > clusters
>>>>>>> > > > and
>>>>>>> > > > > > > >>> mapreduce all in the
one JVM. It is a miracle they
>>>>>>> > > > > > > >>> pass at
>>>>>>> > all.
>>>>>>> > > > > > > Usually
>>>>>>> > > > > > > >>> integration tests
have been cast as unit tests
>>>>>>> > > > > > > >>> because there
>>>>>>> > > was
>>>>>>> > > > no
>>>>>>> > > > > > > where
>>>>>>> > > > > > > >>> else for them to get
an airing.  We have the hbase-it
>>>>>>> > > > > > > >>> suite
>>>>>>> > now
>>>>>>> > > > > which
>>>>>>> > > > > > > would
>>>>>>> > > > > > > >>> be a more apt place
but until these are run on a
>>>>>>> > > > > > > >>> regular
>>>>>>> > basis
>>>>>>> > > in
>>>>>>> > > > > > > public
>>>>>>> > > > > > > >>> for all to see, the
fat integration tests disguised
>>>>>>> > > > > > > >>> as unit
>>>>>>> > > tests
>>>>>>> > > > > > will
>>>>>>> > > > > > > >>> remain.  A review
of our current unit tests weeding
>>>>>>> > > > > > > >>> the old
>>>>>>> > > cruft
>>>>>>> > > > > and
>>>>>>> > > > > > > the
>>>>>>> > > > > > > >>> no longer relevant
or duplicates would be a nice
>>>>>>> > > > > > > >>> undertaking
>>>>>>> > if
>>>>>>> > > > > > > someone is
>>>>>>> > > > > > > >>> looking to contribute.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> Alex Newman has been
working on making our tests work
>>>>>>> > > > > > > >>> up on
>>>>>>> > > > travis
>>>>>>> > > > > > and
>>>>>>> > > > > > > >>> circle-ci.  That'll
be sweet when it goes end-to-end.
>>>>>>> > > > > > > >>> He
>>>>>>> > also
>>>>>>> > > > > added
>>>>>>> > > > > > in
>>>>>>> > > > > > > >>> some "type" categorizations
-- client, filter,
>>>>>>> > > > > > > >>> mapreduce --
>>>>>>> > > > > alongside
>>>>>>> > > > > > > our
>>>>>>> > > > > > > >>> old "sizing" categorizations
of small/medium/large.
>>>>>>> > > > > > > >>> His
>>>>>>> > > thinking
>>>>>>> > > > > is
>>>>>>> > > > > > > that
>>>>>>> > > > > > > >>> we can run these categorizations
in parallel so we
>>>>>>> > > > > > > >>> could run
>>>>>>> > > the
>>>>>>> > > > > > total
>>>>>>> > > > > > > >>> suite in about the
time of the longest test, say
>>>>>>> > 20-30minutes?
>>>>>>> > > > We
>>>>>>> > > > > > > could
>>>>>>> > > > > > > >>> even change Apache
to run them this way.
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>> FYI,
>>>>>>> > > > > > > >>> St.Ack
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>>
>>>>>>> > > > > > > >>
>>>>>>> > > > > > > >
>>>>>>> > > > > > >
>>>>>>> > > > > >
>>>>>>> > > > > >
>>>>>>> > > > > >
>>>>>>> > > > > > --
>>>>>>> > > > > > Sean
>>>>>>> > > > > >
>>>>>>> > > > >
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > --
>>>>>>> > > > Best regards,
>>>>>>> > > >
>>>>>>> > > >    - Andy
>>>>>>> > > >
>>>>>>> > > > Problems worthy of attack prove their worth by
hitting back. -
>>>>>>> > > > Piet
>>>>>>> > Hein
>>>>>>> > > > (via Tom White)
>>>>>>> > > >
>>>>>>> > >
>>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sean
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sean
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sean
>>>
>>>
>>>
>>>
>>> --
>>> Sean
>>
>>
>>
>>
>> --
>> Sean
>
>
>
>
> --
> Sean



-- 
Sean

Mime
View raw message