hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Chauhan <hashut...@apache.org>
Subject Re: Plan: permanently move hive builds from bigtop
Date Thu, 22 May 2014 06:34:17 GMT
Thanks Szehon for your time and contribution in fixing test infra! Looks
like things are good now.


On Wed, May 21, 2014 at 2:09 PM, Szehon Ho <szehon@cloudera.com> wrote:

> The pre-commit builds should be officially good now.  The test logs are
> properly downloaded, and correctly displayed on the HiveQA JIRA comment as
> the link under "TestReport", and all logs themselves in raw form are at
> "Test logs", see HIVE-6937 and HIVE-7080 for details.
>
> Also as we had troubles with the patch-upload trigger job on Apache Jenkins
> having outages and not triggering the test, so we also moved that to the
> Hive Jenkins on the same server:
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/
> ,
> which now runs the hadoopQA script every 10 minutes.
>   I'll continue to monitor this one, as its still new.
>
> Also as some might know already, the Pre-commit build is now using
> -Phadoop-2 and JDK1.7.
>
> Thanks
> Szehon
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/
> >
>
>
> On Tue, Apr 22, 2014 at 4:15 PM, Szehon Ho <szehon@cloudera.com> wrote:
>
> > Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me
> > fix the Jenkins url from Apache-Infra side.  Now there's no need to
> > manually submit Pre-commit builds, unless it got missed somehow.  And in
> > this Jenkins, you can easily see all the JIRA's in the queue:
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/
> >
> > Sorry, please use this URL and not the first one, as job was renamed due
> > to the auto-trigger expecting this format.
> >
> > And to repeat, Test Reporting and the 'Logs located at:' link is still
> > broken and tracked in HIVE-6937.  For now, to see the logs of a failed
> > build, you have to manually navigate.
> > Example, for PreCommit-Hive-BUILD#6, logs are at:
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/
> >.
> > The HiveQA comment on the JIRA should mention the build number.
> >
> > Thanks,
> > Szehon
> >
> >
> > On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho <szehon@cloudera.com> wrote:
> >
> >> It looks great, thanks Lefty!
> >>
> >>
> >> On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz <
> leftyleverenz@gmail.com>wrote:
> >>
> >>> Nice doc, Szehon.  I did some minor editing so you might want to make
> >>> sure
> >>> I didn't introduce any errors.
> >>>
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
> >>>
> >>> -- Lefty
> >>>
> >>>
> >>> On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho <szehon@cloudera.com>
> wrote:
> >>>
> >>> > Migration is done, I updated the wiki to add all the details of the
> new
> >>> > setup:
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
> >>> >
> >>> > New Jenkins URL to submit pre-commit jobs:
> >>> >
> >>> >
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/
> >>> >   Again, this has to be done manually for time being, by clicking on
> >>> 'build
> >>> > with parameters', and entering the issue number as a parameter.  I've
> >>> > submitted some already.  I'll reach out to some committers to get the
> >>> > auto-trigger working.
> >>> >
> >>> > As I mentioned, there is some work to fix the test-reporting, due to
> >>> the
> >>> > framework using old url scheme.  I am tracking it at
> >>> > HIVE-6937<https://issues.apache.org/jira/browse/HIVE-6937>.
> >>> >  For now I am hosting log directory separately, if you want to see
> test
> >>> > logs, you have to manually go the url corresponding to your build,
> >>> like:
> >>> >
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for
> >>> > run#11.  Sorry about that.
> >>> >
> >>> > Let me know if you see other issues, thanks!
> >>> > Szehon
> >>> >
> >>> >
> >>> > On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair <thejas@hortonworks.com
> >
> >>> > wrote:
> >>> >
> >>> > > Sounds good.
> >>> > > Thanks Szehon!
> >>> > >
> >>> > >
> >>> > > On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan <
> >>> hashutosh@apache.org
> >>> > >
> >>> > > wrote:
> >>> > > > +1 Thanks Szehon!
> >>> > > >
> >>> > > >
> >>> > > > On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang <
> xzhang@cloudera.com>
> >>> > > wrote:
> >>> > > >
> >>> > > >> +1. Thanks for taking care of this.
> >>> > > >>
> >>> > > >>
> >>> > > >> On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho <
> szehon@cloudera.com>
> >>> > > wrote:
> >>> > > >>
> >>> > > >> > Hi,
> >>> > > >> >
> >>> > > >> > This week the machine running Hive builds at
> >>> > > >> > http://bigtop01.cloudera.org:8080/view/Hive/?  ran
out of
> >>> space, so
> >>> > > new
> >>> > > >> > jobs like Precommit tests stopped.  Its still not
resolved
> >>> there,
> >>> > > there
> >>> > > >> was
> >>> > > >> > another email today on Bigtop list, but there's
very few
> people
> >>> with
> >>> > > root
> >>> > > >> > access to that host, and they still haven't responded.
> >>> > > >> >
> >>> > > >> > I chatted with Brock, he has also seen various issues
with
> >>> Bigtop
> >>> > > jenkins
> >>> > > >> > in the past, so I am thinking to move the Jenkins
jobs to the
> >>> PTest
> >>> > > >> master
> >>> > > >> > itself, where some PMC already have access and can
admin if
> >>> needed.
> >>> > > >> >  Currently I am hosting the pre-commit Jenkins job
on my own
> EC2
> >>> > > instance
> >>> > > >> > as stop-gap.
> >>> > > >> >
> >>> > > >> > Other advantages of hosting our own Jenkins:
> >>> > > >> > 1. No need to wait for other Bigtop jobs to run.
> >>> > > >> > 2. Bigtop is using a version of Jenkins that doesnt
show
> >>> parameters
> >>> > > like
> >>> > > >> > JIRA number for queued jobs, so impossible to tell
whether a
> >>> patch
> >>> > got
> >>> > > >> > picked up and where it is in queue.
> >>> > > >> > 3. Eliminate network hop from Bigtop box to our
PTest master.
> >>> > > >> >
> >>> > > >> > The disadvantage is:
> >>> > > >> > 1. We don't have much experience doing Jenkins admin,
but it
> >>> doesn't
> >>> > > look
> >>> > > >> > too bad.  Mostly, restart if there's issue and clean
up if out
> >>> of
> >>> > > space.
> >>> > > >> >
> >>> > > >> > I wonder what people think, and if there's any objections
to
> >>> this?
> >>> >  If
> >>> > > >> not,
> >>> > > >> > I'll try setting up this weekend.  Then, there is
some
> follow-up
> >>> > work,
> >>> > > >> like
> >>> > > >> > changing the Jenkins url's displayed in the test
report.
> >>> > > >> >
> >>> > > >> > Thanks!
> >>> > > >> > Szehon
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> > > --
> >>> > > CONFIDENTIALITY NOTICE
> >>> > > NOTICE: This message is intended for the use of the individual
or
> >>> entity
> >>> > to
> >>> > > which it is addressed and may contain information that is
> >>> confidential,
> >>> > > privileged and exempt from disclosure under applicable law. If
the
> >>> reader
> >>> > > of this message is not the intended recipient, you are hereby
> >>> notified
> >>> > that
> >>> > > any printing, copying, dissemination, distribution, disclosure
or
> >>> > > forwarding of this communication is strictly prohibited. If you
> have
> >>> > > received this communication in error, please contact the sender
> >>> > immediately
> >>> > > and delete it from your system. Thank You.
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message