hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)
Date Tue, 03 Nov 2015 07:03:06 GMT
I just did an edit of all of our jenkins configuration. I changed the
post-build zombie section. The surefire-killer was hanging out here it
seems (For detail, see HBASE-14589 Looking for the surefire-killer; builds
being killed...). I may have messed up builds. Will fix in morning if so
(currently builds seem a bit backed up so won't wait around).

St.Ack

On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey <busbey@cloudera.com> wrote:

> Hi Folks,
>
> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS]
> reducing our load on builds.a.o' on reasoning or if you have concerns.
>
> Our project job view now has a brief summary of what's going on:
>
> https://builds.apache.org/view/H-L/view/HBase/
>
> * Where we had matrix builds, I confirmed they're doing the same thing
> (in at least one test-configuration) as the non-matrix version
> * I disabled jobs that were duplicating work done in matrix jobs, or
> that only worked on a tag
> * Made sure all non-disabled 0.98+ jobs are set to notify jira +
> builds mailing list
> * I updated pre-1.1 jobs to look for changes 1/day
> * I updated 1.1 jobs to look for changes every 8 hours
> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours
>
> There's still some non-pressing clean up work to do, so I'll file a
> jira later today just to make sure it's all recorded somewhere.
>
> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <busbey@cloudera.com> wrote:
> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2
> builds to
> > use branch-1.2. (ref HBASE-13912)
> >
> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >>
> >> I've add a new build test to run the ITs under sunny day conditions
> using
> >> minicluster for both java 7 and java 8 on the 1.2 line.
> >>
> >> https://builds.apache.org/job/HBase-1.2-IT/
> >>
> >> I haven't turned on notifications yet, because BulkIngest and TestIngest
> >> are read. details on HBASE-13750
> >>
> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >>>
> >>> sigh. that should have been "is now a matrix build".
> >>>
> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >>>>
> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8
> in
> >>>> parallel.
> >>>>
> >>>> I saved the old job for now and left it disabled[2].
> >>>>
> >>>>
> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/
> >>>> [2]:
> >>>>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/
> >>>>
> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <busbey@cloudera.com>
> >>>> wrote:
> >>>>>
> >>>>> Sorry for the noise. I also updated the checkstyle step on
> HBase-TRUNK
> >>>>>
> >>>>> from
> >>>>>     mvn checkstyle:checkstyle
> >>>>> to
> >>>>>     mvn -DskipTests package checkstyle:checkstyle
> >>>>>
> >>>>> to deal with the same issue. Looks all clear now.
> >>>>>
> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <busbey@cloudera.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Updated the following builds:
> >>>>>>
> >>>>>> * HBase-TRUNK
> >>>>>>
> >>>>>> moved from
> >>>>>>
> >>>>>>     mvn clean compile findbugs:findbugs
> >>>>>>
> >>>>>> to
> >>>>>>
> >>>>>>    mvn clean -DskipTests package findbugs:findbugs
> >>>>>>
> >>>>>> To work around the known issue where we can't do a bootstrap
compile
> >>>>>> without previous install or remote SNAPSHOT artifacts. (recently
> triggered
> >>>>>> by the addition of the procedure module on master)
> >>>>>>
> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino219@gmail.com>
wrote:
> >>>>>>>
> >>>>>>> Make findbugs and checkstyle plugins always run.
> >>>>>>> The default behavior only publishes result for stable builds,
but
> >>>>>>> master is
> >>>>>>> not always stable and sometimes we introduce new warnings
in
> unstable
> >>>>>>> builds(see builds 6310-6314).
> >>>>>>> And findbugs and checkstyle will not fail unless we abort
the
> >>>>>>> building
> >>>>>>> process I think, so it is safe to turn it on every time.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> >>>>>>>
> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report.
> >>>>>>> >
> >>>>>>> > Cheers
> >>>>>>> >
> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino219@gmail.com>
> wrote:
> >>>>>>> >
> >>>>>>> > > Going to change the config of HBase-TRUNK jenkins
to show
> >>>>>>> > > findbugs,
> >>>>>>> > > checkstyle and jacoco coverage report.
> >>>>>>> > > The config has been tested on
> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco
for nearly 30
> >>>>>>> > > times.
> >>>>>>> > Not
> >>>>>>> > > much different from HBase-TRUNK unless it runs
~30% slower(the
> >>>>>>> > > overhead
> >>>>>>> > of
> >>>>>>> > > collecting information for code coverage).
> >>>>>>> > > Thanks.
> >>>>>>> > >
> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <apurtell@apache.org
> >:
> >>>>>>> > >
> >>>>>>> > > > +1, thanks a lot for improving our build
hygiene.
> >>>>>>> > > >
> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar
> >>>>>>> > > > <enis.soz@gmail.com>
> >>>>>>> > > wrote:
> >>>>>>> > > >
> >>>>>>> > > > > Thanks Sean. This is great.
> >>>>>>> > > > >
> >>>>>>> > > > > Enis
> >>>>>>> > > > >
> >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean
Busbey
> >>>>>>> > > > > <busbey@cloudera.com>
> >>>>>>> > > > wrote:
> >>>>>>> > > > >
> >>>>>>> > > > > > FYI I just finished chasing down
the breakage for mvn
> site
> >>>>>>> > > > > > on all
> >>>>>>> > > patch
> >>>>>>> > > > > > builds.
> >>>>>>> > > > > >
> >>>>>>> > > > > > HBASE-13191 consolidates the few
places in the test-patch
> >>>>>>> > > > > > code
> >>>>>>> > where
> >>>>>>> > > we
> >>>>>>> > > > > > hard coded MAVEN_OPTS.
> >>>>>>> > > > > >
> >>>>>>> > > > > > If you look at the PreCommit job
now, we use the "set
> >>>>>>> > > > > > environment
> >>>>>>> > > > > > variables" option to set MAVEN_OPTS
and then everything
> >>>>>>> > > > > > else
> >>>>>>> > respects
> >>>>>>> > > > > that
> >>>>>>> > > > > > setting.
> >>>>>>> > > > > >
> >>>>>>> > > > > > I set the initial value to be a
combination of the memory
> >>>>>>> > limitations
> >>>>>>> > > > > we've
> >>>>>>> > > > > > been actually running with (the
~6G was getting ignored)
> >>>>>>> > > > > > and the
> >>>>>>> > > > permgen
> >>>>>>> > > > > > needed for site.
> >>>>>>> > > > > >
> >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData
> -XX:MaxPermSize=256m
> >>>>>>> > > > > >
> >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM,
Stack <
> stack@duboce.net>
> >>>>>>> > > > > > wrote:
> >>>>>>> > > > > >
> >>>>>>> > > > > > > I just made TRUNK and branch-1
builds use same jvm as
> >>>>>>> > > > > > > patch-build
> >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51
-- and I set the MAVEN_OPTS
> to
> >>>>>>> > > > > > > be the
> >>>>>>> > > same
> >>>>>>> > > > as
> >>>>>>> > > > > > > those of trunk build too,
setting
> >>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"...
it
> >>>>>>> > had
> >>>>>>> > > > been
> >>>>>>> > > > > > > 3000.
> >>>>>>> > > > > > >
> >>>>>>> > > > > > > Yours,
> >>>>>>> > > > > > > St.Ack
> >>>>>>> > > > > > >
> >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22
PM, Stack <
> stack@duboce.net>
> >>>>>>> > > > > > > wrote:
> >>>>>>> > > > > > >
> >>>>>>> > > > > > > > I upped hadoopqa retention
to keep last 100 builds
> and
> >>>>>>> > > > > > > > or last
> >>>>>>> > 7
> >>>>>>> > > > > days,
> >>>>>>> > > > > > > > whichever comes first.
> >>>>>>> > > > > > > > St.Ack
> >>>>>>> > > > > > > >
> >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at
9:38 AM, Stack
> >>>>>>> > > > > > > > <stack@duboce.net>
> >>>>>>> > wrote:
> >>>>>>> > > > > > > >
> >>>>>>> > > > > > > >> Branch-1 and master
have stabilized and now run
> mostly
> >>>>>>> > > > > > > >> blue
> >>>>>>> > > (give
> >>>>>>> > > > or
> >>>>>>> > > > > > > take
> >>>>>>> > > > > > > >> the odd failure)
[1][2]. Having a mostly blue
> branch-1
> >>>>>>> > > > > > > >> has
> >>>>>>> > > helped
> >>>>>>> > > > us
> >>>>>>> > > > > > > >> identify at least
one destabilizing commit in the
> last
> >>>>>>> > > > > > > >> few
> >>>>>>> > days,
> >>>>>>> > > > > maybe
> >>>>>>> > > > > > > two;
> >>>>>>> > > > > > > >> this is as it should
be (smile).
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> Lets keep our builds
blue. If you commit a patch,
> make
> >>>>>>> > > > > > > >> sure
> >>>>>>> > > > > subsequent
> >>>>>>> > > > > > > >> builds stay blue.
You can subscribe to
> >>>>>>> > builds@hbase.apache.org
> >>>>>>> > > to
> >>>>>>> > > > > get
> >>>>>>> > > > > > > >> notice of failures
if not already subscribed.
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> Thanks,
> >>>>>>> > > > > > > >> St.Ack
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> 1.
> >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
> >>>>>>> > > > > > > >> 2.
> >>>>>>> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014
at 4:41 PM, Stack
> >>>>>>> > > > > > > >> <stack@duboce.net>
> >>>>>>> > > wrote:
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >>> A few notes on
testing.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Too long to read,
infra is more capable now and
> after
> >>>>>>> > > > > > > >>> some
> >>>>>>> > > work,
> >>>>>>> > > > we
> >>>>>>> > > > > > are
> >>>>>>> > > > > > > >>> seeing branch-1
and trunk mostly running blue. Lets
> >>>>>>> > > > > > > >>> try and
> >>>>>>> > > keep
> >>>>>>> > > > it
> >>>>>>> > > > > > > this
> >>>>>>> > > > > > > >>> way going forward.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Apache Infra
has new, more capable hardware.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> A recent spurt
of test fixing combined with more
> >>>>>>> > > > > > > >>> capable
> >>>>>>> > > hardware
> >>>>>>> > > > > > seems
> >>>>>>> > > > > > > >>> to have gotten
us to a new place; tests are mostly
> >>>>>>> > > > > > > >>> passing
> >>>>>>> > now
> >>>>>>> > > on
> >>>>>>> > > > > > > branch-1
> >>>>>>> > > > > > > >>> and master. 
Lets try and keep it this way and
> start
> >>>>>>> > > > > > > >>> to trust
> >>>>>>> > > our
> >>>>>>> > > > > > test
> >>>>>>> > > > > > > runs
> >>>>>>> > > > > > > >>> again.  Just
a few flakies remain.  Lets try and
> nail
> >>>>>>> > > > > > > >>> them.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Our tests now
run in parallel with other test
> suites
> >>>>>>> > > > > > > >>> where
> >>>>>>> > > > previous
> >>>>>>> > > > > > we
> >>>>>>> > > > > > > >>> ran alone. You
can see this sometimes when our
> zombie
> >>>>>>> > detector
> >>>>>>> > > > > > reports
> >>>>>>> > > > > > > >>> tests from another
project altogether as lingerers
> >>>>>>> > > > > > > >>> (To be
> >>>>>>> > > fixed).
> >>>>>>> > > > > > > Some of
> >>>>>>> > > > > > > >>> our tests are
failing because a concurrent hbase
> run
> >>>>>>> > > > > > > >>> is
> >>>>>>> > undoing
> >>>>>>> > > > > > > classes and
> >>>>>>> > > > > > > >>> data from under
it. Also, lets fix.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Our tests are
brittle. It takes 75minutes for them
> to
> >>>>>>> > complete.
> >>>>>>> > > > > Many
> >>>>>>> > > > > > > >>> are heavy-duty
integration tests starting up
> multiple
> >>>>>>> > clusters
> >>>>>>> > > > and
> >>>>>>> > > > > > > >>> mapreduce all
in the one JVM. It is a miracle they
> >>>>>>> > > > > > > >>> pass at
> >>>>>>> > all.
> >>>>>>> > > > > > > Usually
> >>>>>>> > > > > > > >>> integration tests
have been cast as unit tests
> >>>>>>> > > > > > > >>> because there
> >>>>>>> > > was
> >>>>>>> > > > no
> >>>>>>> > > > > > > where
> >>>>>>> > > > > > > >>> else for them
to get an airing.  We have the
> hbase-it
> >>>>>>> > > > > > > >>> suite
> >>>>>>> > now
> >>>>>>> > > > > which
> >>>>>>> > > > > > > would
> >>>>>>> > > > > > > >>> be a more apt
place but until these are run on a
> >>>>>>> > > > > > > >>> regular
> >>>>>>> > basis
> >>>>>>> > > in
> >>>>>>> > > > > > > public
> >>>>>>> > > > > > > >>> for all to see,
the fat integration tests disguised
> >>>>>>> > > > > > > >>> as unit
> >>>>>>> > > tests
> >>>>>>> > > > > > will
> >>>>>>> > > > > > > >>> remain.  A review
of our current unit tests weeding
> >>>>>>> > > > > > > >>> the old
> >>>>>>> > > cruft
> >>>>>>> > > > > and
> >>>>>>> > > > > > > the
> >>>>>>> > > > > > > >>> no longer relevant
or duplicates would be a nice
> >>>>>>> > > > > > > >>> undertaking
> >>>>>>> > if
> >>>>>>> > > > > > > someone is
> >>>>>>> > > > > > > >>> looking to contribute.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Alex Newman has
been working on making our tests
> work
> >>>>>>> > > > > > > >>> up on
> >>>>>>> > > > travis
> >>>>>>> > > > > > and
> >>>>>>> > > > > > > >>> circle-ci.  That'll
be sweet when it goes
> end-to-end.
> >>>>>>> > > > > > > >>> He
> >>>>>>> > also
> >>>>>>> > > > > added
> >>>>>>> > > > > > in
> >>>>>>> > > > > > > >>> some "type" categorizations
-- client, filter,
> >>>>>>> > > > > > > >>> mapreduce --
> >>>>>>> > > > > alongside
> >>>>>>> > > > > > > our
> >>>>>>> > > > > > > >>> old "sizing"
categorizations of small/medium/large.
> >>>>>>> > > > > > > >>> His
> >>>>>>> > > thinking
> >>>>>>> > > > > is
> >>>>>>> > > > > > > that
> >>>>>>> > > > > > > >>> we can run these
categorizations in parallel so we
> >>>>>>> > > > > > > >>> could run
> >>>>>>> > > the
> >>>>>>> > > > > > total
> >>>>>>> > > > > > > >>> suite in about
the time of the longest test, say
> >>>>>>> > 20-30minutes?
> >>>>>>> > > > We
> >>>>>>> > > > > > > could
> >>>>>>> > > > > > > >>> even change Apache
to run them this way.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> FYI,
> >>>>>>> > > > > > > >>> St.Ack
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >
> >>>>>>> > > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > > --
> >>>>>>> > > > > > Sean
> >>>>>>> > > > > >
> >>>>>>> > > > >
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > > --
> >>>>>>> > > > Best regards,
> >>>>>>> > > >
> >>>>>>> > > >    - Andy
> >>>>>>> > > >
> >>>>>>> > > > Problems worthy of attack prove their worth
by hitting back.
> -
> >>>>>>> > > > Piet
> >>>>>>> > Hein
> >>>>>>> > > > (via Tom White)
> >>>>>>> > > >
> >>>>>>> > >
> >>>>>>> >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Sean
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Sean
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sean
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Sean
> >>
> >>
> >>
> >>
> >> --
> >> Sean
> >
> >
> >
> >
> > --
> > Sean
>
>
>
> --
> Sean
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message