hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)
Date Sat, 20 Jun 2015 00:50:51 GMT
Please see this note on hadoop-common about an infra issue that's causing a
temporary delay:

*http://s.apache.org/6XU <http://s.apache.org/6XU>*

On Fri, Jun 19, 2015 at 7:31 PM, Enis Söztutar <enis.soz@gmail.com> wrote:

> It seems that precommit-admin job which triggers the precommit-hbase job is
> not running as frequently as it used to be. Only once every ~6 hours or so.
>
> https://builds.apache.org/job/PreCommit-Admin/
>
> The job is starved for the slaves.
>
> Enis
>
> On Fri, Jun 19, 2015 at 5:23 PM, Enis Söztutar <enis.soz@gmail.com> wrote:
>
> > hadoopqa is acting weird for the last couple of days. It does not report
> > back the results. One example is:
> > https://builds.apache.org/job/PreCommit-HBASE-Build/14475/console.
> >
> > I've excluded ubuntu3 from the list because it does not have protoc-2.5.
> >
> >
> > On Mon, Jun 15, 2015 at 9:55 PM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >
> >> Added new HBase-1.3 build or branch-1 and changed the existing 1.2
> builds
> >> to use branch-1.2. (ref HBASE-13912)
> >>
> >> On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >>
> >> > I've add a new build test to run the ITs under sunny day conditions
> >> using
> >> > minicluster for both java 7 and java 8 on the 1.2 line.
> >> >
> >> > https://builds.apache.org/job/HBase-1.2-IT/
> >> >
> >> > I haven't turned on notifications yet, because BulkIngest and
> TestIngest
> >> > are read. details on HBASE-13750
> >> >
> >> > On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <busbey@cloudera.com>
> >> wrote:
> >> >
> >> >> sigh. that should have been "is now a matrix build".
> >> >>
> >> >> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <busbey@cloudera.com>
> >> wrote:
> >> >>
> >> >>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and
JDK8
> >> in
> >> >>> parallel.
> >> >>>
> >> >>> I saved the old job for now and left it disabled[2].
> >> >>>
> >> >>>
> >> >>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/
> >> >>> [2]:
> >> >>>
> >>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/
> >> >>>
> >> >>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <busbey@cloudera.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Sorry for the noise. I also updated the checkstyle step on
> >> HBase-TRUNK
> >> >>>>
> >> >>>> from
> >> >>>>     mvn checkstyle:checkstyle
> >> >>>> to
> >> >>>>     mvn -DskipTests package checkstyle:checkstyle
> >> >>>>
> >> >>>> to deal with the same issue. Looks all clear now.
> >> >>>>
> >> >>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <busbey@cloudera.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> Updated the following builds:
> >> >>>>>
> >> >>>>> * HBase-TRUNK
> >> >>>>>
> >> >>>>> moved from
> >> >>>>>
> >> >>>>>     mvn clean compile findbugs:findbugs
> >> >>>>>
> >> >>>>> to
> >> >>>>>
> >> >>>>>    mvn clean -DskipTests package findbugs:findbugs
> >> >>>>>
> >> >>>>> To work around the known issue where we can't do a bootstrap
> compile
> >> >>>>> without previous install or remote SNAPSHOT artifacts.
(recently
> >> triggered
> >> >>>>> by the addition of the procedure module on master)
> >> >>>>>
> >> >>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino219@gmail.com>
> wrote:
> >> >>>>>
> >> >>>>>> Make findbugs and checkstyle plugins always run.
> >> >>>>>> The default behavior only publishes result for stable
builds, but
> >> >>>>>> master is
> >> >>>>>> not always stable and sometimes we introduce new warnings
in
> >> unstable
> >> >>>>>> builds(see builds 6310-6314).
> >> >>>>>> And findbugs and checkstyle will not fail unless we
abort the
> >> building
> >> >>>>>> process I think, so it is safe to turn it on every
time.
> >> >>>>>>
> >> >>>>>> Thanks.
> >> >>>>>>
> >> >>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> >> >>>>>>
> >> >>>>>> > +1 on letting HBase-TRUNK jenkins show coverage
report.
> >> >>>>>> >
> >> >>>>>> > Cheers
> >> >>>>>> >
> >> >>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino219@gmail.com>
> >> wrote:
> >> >>>>>> >
> >> >>>>>> > > Going to change the config of HBase-TRUNK
jenkins to show
> >> >>>>>> findbugs,
> >> >>>>>> > > checkstyle and jacoco coverage report.
> >> >>>>>> > > The config has been tested on
> >> >>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco
for nearly
> 30
> >> >>>>>> times.
> >> >>>>>> > Not
> >> >>>>>> > > much different from HBase-TRUNK unless it
runs ~30%
> slower(the
> >> >>>>>> overhead
> >> >>>>>> > of
> >> >>>>>> > > collecting information for code coverage).
> >> >>>>>> > > Thanks.
> >> >>>>>> > >
> >> >>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell
<
> apurtell@apache.org
> >> >:
> >> >>>>>> > >
> >> >>>>>> > > > +1, thanks a lot for improving our build
hygiene.
> >> >>>>>> > > >
> >> >>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis
Söztutar <
> >> >>>>>> enis.soz@gmail.com>
> >> >>>>>> > > wrote:
> >> >>>>>> > > >
> >> >>>>>> > > > > Thanks Sean. This is great.
> >> >>>>>> > > > >
> >> >>>>>> > > > > Enis
> >> >>>>>> > > > >
> >> >>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM,
Sean Busbey <
> >> >>>>>> busbey@cloudera.com>
> >> >>>>>> > > > wrote:
> >> >>>>>> > > > >
> >> >>>>>> > > > > > FYI I just finished chasing
down the breakage for mvn
> >> site
> >> >>>>>> on all
> >> >>>>>> > > patch
> >> >>>>>> > > > > > builds.
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > HBASE-13191 consolidates the
few places in the
> test-patch
> >> >>>>>> code
> >> >>>>>> > where
> >> >>>>>> > > we
> >> >>>>>> > > > > > hard coded MAVEN_OPTS.
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > If you look at the PreCommit
job now, we use the "set
> >> >>>>>> environment
> >> >>>>>> > > > > > variables" option to set MAVEN_OPTS
and then everything
> >> else
> >> >>>>>> > respects
> >> >>>>>> > > > > that
> >> >>>>>> > > > > > setting.
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > I set the initial value to
be a combination of the
> memory
> >> >>>>>> > limitations
> >> >>>>>> > > > > we've
> >> >>>>>> > > > > > been actually running with
(the ~6G was getting
> ignored)
> >> >>>>>> and the
> >> >>>>>> > > > permgen
> >> >>>>>> > > > > > needed for site.
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData
> >> -XX:MaxPermSize=256m
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46
PM, Stack <
> >> stack@duboce.net>
> >> >>>>>> wrote:
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > > I just made TRUNK and
branch-1 builds use same jvm as
> >> >>>>>> patch-build
> >> >>>>>> > > > > > > (hadoopqa) -- i.e. jdku51
-- and I set the MAVEN_OPTS
> >> to
> >> >>>>>> be the
> >> >>>>>> > > same
> >> >>>>>> > > > as
> >> >>>>>> > > > > > > those of trunk build
too, setting
> >> >>>>>> MAVEN_OPTS="-Xmx6100m"... it
> >> >>>>>> > had
> >> >>>>>> > > > been
> >> >>>>>> > > > > > > 3000.
> >> >>>>>> > > > > > >
> >> >>>>>> > > > > > > Yours,
> >> >>>>>> > > > > > > St.Ack
> >> >>>>>> > > > > > >
> >> >>>>>> > > > > > > On Wed, Dec 31, 2014
at 4:22 PM, Stack <
> >> stack@duboce.net>
> >> >>>>>> wrote:
> >> >>>>>> > > > > > >
> >> >>>>>> > > > > > > > I upped hadoopqa
retention to keep last 100 builds
> >> and
> >> >>>>>> or last
> >> >>>>>> > 7
> >> >>>>>> > > > > days,
> >> >>>>>> > > > > > > > whichever comes
first.
> >> >>>>>> > > > > > > > St.Ack
> >> >>>>>> > > > > > > >
> >> >>>>>> > > > > > > > On Tue, Nov 4, 2014
at 9:38 AM, Stack <
> >> stack@duboce.net
> >> >>>>>> >
> >> >>>>>> > wrote:
> >> >>>>>> > > > > > > >
> >> >>>>>> > > > > > > >> Branch-1 and
master have stabilized and now run
> >> mostly
> >> >>>>>> blue
> >> >>>>>> > > (give
> >> >>>>>> > > > or
> >> >>>>>> > > > > > > take
> >> >>>>>> > > > > > > >> the odd failure)
[1][2]. Having a mostly blue
> >> branch-1
> >> >>>>>> has
> >> >>>>>> > > helped
> >> >>>>>> > > > us
> >> >>>>>> > > > > > > >> identify at
least one destabilizing commit in the
> >> last
> >> >>>>>> few
> >> >>>>>> > days,
> >> >>>>>> > > > > maybe
> >> >>>>>> > > > > > > two;
> >> >>>>>> > > > > > > >> this is as it
should be (smile).
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >> Lets keep our
builds blue. If you commit a patch,
> >> make
> >> >>>>>> sure
> >> >>>>>> > > > > subsequent
> >> >>>>>> > > > > > > >> builds stay
blue. You can subscribe to
> >> >>>>>> > builds@hbase.apache.org
> >> >>>>>> > > to
> >> >>>>>> > > > > get
> >> >>>>>> > > > > > > >> notice of failures
if not already subscribed.
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >> Thanks,
> >> >>>>>> > > > > > > >> St.Ack
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >> 1.
> >> >>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
> >> >>>>>> > > > > > > >> 2.
> >> >>>>>> > >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >> On Mon, Oct
13, 2014 at 4:41 PM, Stack <
> >> >>>>>> stack@duboce.net>
> >> >>>>>> > > wrote:
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >>> A few notes
on testing.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> Too long
to read, infra is more capable now and
> >> after
> >> >>>>>> some
> >> >>>>>> > > work,
> >> >>>>>> > > > we
> >> >>>>>> > > > > > are
> >> >>>>>> > > > > > > >>> seeing branch-1
and trunk mostly running blue.
> Lets
> >> >>>>>> try and
> >> >>>>>> > > keep
> >> >>>>>> > > > it
> >> >>>>>> > > > > > > this
> >> >>>>>> > > > > > > >>> way going
forward.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> Apache Infra
has new, more capable hardware.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> A recent
spurt of test fixing combined with more
> >> >>>>>> capable
> >> >>>>>> > > hardware
> >> >>>>>> > > > > > seems
> >> >>>>>> > > > > > > >>> to have
gotten us to a new place; tests are
> mostly
> >> >>>>>> passing
> >> >>>>>> > now
> >> >>>>>> > > on
> >> >>>>>> > > > > > > branch-1
> >> >>>>>> > > > > > > >>> and master.
 Lets try and keep it this way and
> >> start
> >> >>>>>> to trust
> >> >>>>>> > > our
> >> >>>>>> > > > > > test
> >> >>>>>> > > > > > > runs
> >> >>>>>> > > > > > > >>> again. 
Just a few flakies remain.  Lets try and
> >> nail
> >> >>>>>> them.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> Our tests
now run in parallel with other test
> >> suites
> >> >>>>>> where
> >> >>>>>> > > > previous
> >> >>>>>> > > > > > we
> >> >>>>>> > > > > > > >>> ran alone.
You can see this sometimes when our
> >> zombie
> >> >>>>>> > detector
> >> >>>>>> > > > > > reports
> >> >>>>>> > > > > > > >>> tests from
another project altogether as
> lingerers
> >> >>>>>> (To be
> >> >>>>>> > > fixed).
> >> >>>>>> > > > > > > Some of
> >> >>>>>> > > > > > > >>> our tests
are failing because a concurrent hbase
> >> run
> >> >>>>>> is
> >> >>>>>> > undoing
> >> >>>>>> > > > > > > classes and
> >> >>>>>> > > > > > > >>> data from
under it. Also, lets fix.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> Our tests
are brittle. It takes 75minutes for
> them
> >> to
> >> >>>>>> > complete.
> >> >>>>>> > > > > Many
> >> >>>>>> > > > > > > >>> are heavy-duty
integration tests starting up
> >> multiple
> >> >>>>>> > clusters
> >> >>>>>> > > > and
> >> >>>>>> > > > > > > >>> mapreduce
all in the one JVM. It is a miracle
> they
> >> >>>>>> pass at
> >> >>>>>> > all.
> >> >>>>>> > > > > > > Usually
> >> >>>>>> > > > > > > >>> integration
tests have been cast as unit tests
> >> >>>>>> because there
> >> >>>>>> > > was
> >> >>>>>> > > > no
> >> >>>>>> > > > > > > where
> >> >>>>>> > > > > > > >>> else for
them to get an airing.  We have the
> >> hbase-it
> >> >>>>>> suite
> >> >>>>>> > now
> >> >>>>>> > > > > which
> >> >>>>>> > > > > > > would
> >> >>>>>> > > > > > > >>> be a more
apt place but until these are run on a
> >> >>>>>> regular
> >> >>>>>> > basis
> >> >>>>>> > > in
> >> >>>>>> > > > > > > public
> >> >>>>>> > > > > > > >>> for all
to see, the fat integration tests
> disguised
> >> >>>>>> as unit
> >> >>>>>> > > tests
> >> >>>>>> > > > > > will
> >> >>>>>> > > > > > > >>> remain.
 A review of our current unit tests
> weeding
> >> >>>>>> the old
> >> >>>>>> > > cruft
> >> >>>>>> > > > > and
> >> >>>>>> > > > > > > the
> >> >>>>>> > > > > > > >>> no longer
relevant or duplicates would be a nice
> >> >>>>>> undertaking
> >> >>>>>> > if
> >> >>>>>> > > > > > > someone is
> >> >>>>>> > > > > > > >>> looking
to contribute.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> Alex Newman
has been working on making our tests
> >> work
> >> >>>>>> up on
> >> >>>>>> > > > travis
> >> >>>>>> > > > > > and
> >> >>>>>> > > > > > > >>> circle-ci.
 That'll be sweet when it goes
> >> >>>>>> end-to-end.  He
> >> >>>>>> > also
> >> >>>>>> > > > > added
> >> >>>>>> > > > > > in
> >> >>>>>> > > > > > > >>> some "type"
categorizations -- client, filter,
> >> >>>>>> mapreduce --
> >> >>>>>> > > > > alongside
> >> >>>>>> > > > > > > our
> >> >>>>>> > > > > > > >>> old "sizing"
categorizations of
> small/medium/large.
> >> >>>>>> His
> >> >>>>>> > > thinking
> >> >>>>>> > > > > is
> >> >>>>>> > > > > > > that
> >> >>>>>> > > > > > > >>> we can run
these categorizations in parallel so
> we
> >> >>>>>> could run
> >> >>>>>> > > the
> >> >>>>>> > > > > > total
> >> >>>>>> > > > > > > >>> suite in
about the time of the longest test, say
> >> >>>>>> > 20-30minutes?
> >> >>>>>> > > > We
> >> >>>>>> > > > > > > could
> >> >>>>>> > > > > > > >>> even change
Apache to run them this way.
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>> FYI,
> >> >>>>>> > > > > > > >>> St.Ack
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>>
> >> >>>>>> > > > > > > >>
> >> >>>>>> > > > > > > >
> >> >>>>>> > > > > > >
> >> >>>>>> > > > > >
> >> >>>>>> > > > > >
> >> >>>>>> > > > > >
> >> >>>>>> > > > > > --
> >> >>>>>> > > > > > Sean
> >> >>>>>> > > > > >
> >> >>>>>> > > > >
> >> >>>>>> > > >
> >> >>>>>> > > >
> >> >>>>>> > > >
> >> >>>>>> > > > --
> >> >>>>>> > > > Best regards,
> >> >>>>>> > > >
> >> >>>>>> > > >    - Andy
> >> >>>>>> > > >
> >> >>>>>> > > > Problems worthy of attack prove their
worth by hitting
> back.
> >> -
> >> >>>>>> Piet
> >> >>>>>> > Hein
> >> >>>>>> > > > (via Tom White)
> >> >>>>>> > > >
> >> >>>>>> > >
> >> >>>>>> >
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> Sean
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Sean
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Sean
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Sean
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Sean
> >> >
> >>
> >>
> >>
> >> --
> >> Sean
> >>
> >
> >
>



-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message