Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of szehon@cloudera.com
 designates 74.125.82.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPs2cbE6GFcsR3z_a2=bSb_1XKTnitGVWG4Xst+HE6KF=1QVmQ@mail.gmail.com>
References: 
 <CAPs2cbEQO_Xzc51e3xEJkceuY8Hyr8EHoqM04hrxns_nWciitw@mail.gmail.com>
 <CADx-ob3BbSNUOBGi6RuM7Wr4JSN-jXT22AB+deaDkiwF35e9hg@mail.gmail.com>
 <CA+FBdFQn+9VDUOQg2Vkybm7Oy1eg-aDhqcJau0-8UEC+Z2JQ8A@mail.gmail.com>
 <CABgNGzfv-sv0NUM_gG3qPp=eSsiSb-_tTiJOvSYGDazaUPT=AQ@mail.gmail.com>
 <CAPs2cbFpe+Hfac=--4Rj6NKdM05fUTomTSR3wYYHWksWp_Fzkw@mail.gmail.com>
 <CACEcauU19_GOaQjVaikDkLTyF88s8NZCUtmZXBUwEe+QST8LbQ@mail.gmail.com>
 <CAPs2cbE6GFcsR3z_a2=bSb_1XKTnitGVWG4Xst+HE6KF=1QVmQ@mail.gmail.com>
From: Szehon Ho <szehon@cloudera.com>
Date: Tue, 22 Apr 2014 16:15:22 -0700
Message-ID: 
 <CAPs2cbFJqK+Qae7w_7QyQMyXPhWGRYh7O6qq7pXzN1yqZg+yMg@mail.gmail.com>
Subject: Re: Plan: permanently move hive builds from bigtop
To: "dev@hive.apache.org" <dev@hive.apache.org>
Content-Type: multipart/alternative; boundary=f46d04138e5f82151f04f7a9cb47

--f46d04138e5f82151f04f7a9cb47
Content-Type: text/plain; charset=ISO-8859-1

Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me
fix the Jenkins url from Apache-Infra side.  Now there's no need to
manually submit Pre-commit builds, unless it got missed somehow.  And in
this Jenkins, you can easily see all the JIRA's in the queue:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/

Sorry, please use this URL and not the first one, as job was renamed due to
the auto-trigger expecting this format.

And to repeat, Test Reporting and the 'Logs located at:' link is still
broken and tracked in HIVE-6937.  For now, to see the logs of a failed
build, you have to manually navigate.
Example, for PreCommit-Hive-BUILD#6, logs are at:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/<http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/>.
The HiveQA comment on the JIRA should mention the build number.

Thanks,
Szehon


On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho <szehon@cloudera.com> wrote:

> It looks great, thanks Lefty!
>
>
> On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz <leftyleverenz@gmail.com>wrote:
>
>> Nice doc, Szehon.  I did some minor editing so you might want to make sure
>> I didn't introduce any errors.
>>
>>
>> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
>>
>> -- Lefty
>>
>>
>> On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho <szehon@cloudera.com> wrote:
>>
>> > Migration is done, I updated the wiki to add all the details of the new
>> > setup:
>> >
>> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
>> >
>> > New Jenkins URL to submit pre-commit jobs:
>> >
>> >
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/
>> >   Again, this has to be done manually for time being, by clicking on
>> 'build
>> > with parameters', and entering the issue number as a parameter.  I've
>> > submitted some already.  I'll reach out to some committers to get the
>> > auto-trigger working.
>> >
>> > As I mentioned, there is some work to fix the test-reporting, due to the
>> > framework using old url scheme.  I am tracking it at
>> > HIVE-6937<https://issues.apache.org/jira/browse/HIVE-6937>.
>> >  For now I am hosting log directory separately, if you want to see test
>> > logs, you have to manually go the url corresponding to your build, like:
>> >
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for
>> > run#11.  Sorry about that.
>> >
>> > Let me know if you see other issues, thanks!
>> > Szehon
>> >
>> >
>> > On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair <thejas@hortonworks.com>
>> > wrote:
>> >
>> > > Sounds good.
>> > > Thanks Szehon!
>> > >
>> > >
>> > > On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan <
>> hashutosh@apache.org
>> > >
>> > > wrote:
>> > > > +1 Thanks Szehon!
>> > > >
>> > > >
>> > > > On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang <xzhang@cloudera.com>
>> > > wrote:
>> > > >
>> > > >> +1. Thanks for taking care of this.
>> > > >>
>> > > >>
>> > > >> On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho <szehon@cloudera.com>
>> > > wrote:
>> > > >>
>> > > >> > Hi,
>> > > >> >
>> > > >> > This week the machine running Hive builds at
>> > > >> > http://bigtop01.cloudera.org:8080/view/Hive/?  ran out of
>> space, so
>> > > new
>> > > >> > jobs like Precommit tests stopped.  Its still not resolved there,
>> > > there
>> > > >> was
>> > > >> > another email today on Bigtop list, but there's very few people
>> with
>> > > root
>> > > >> > access to that host, and they still haven't responded.
>> > > >> >
>> > > >> > I chatted with Brock, he has also seen various issues with Bigtop
>> > > jenkins
>> > > >> > in the past, so I am thinking to move the Jenkins jobs to the
>> PTest
>> > > >> master
>> > > >> > itself, where some PMC already have access and can admin if
>> needed.
>> > > >> >  Currently I am hosting the pre-commit Jenkins job on my own EC2
>> > > instance
>> > > >> > as stop-gap.
>> > > >> >
>> > > >> > Other advantages of hosting our own Jenkins:
>> > > >> > 1. No need to wait for other Bigtop jobs to run.
>> > > >> > 2. Bigtop is using a version of Jenkins that doesnt show
>> parameters
>> > > like
>> > > >> > JIRA number for queued jobs, so impossible to tell whether a
>> patch
>> > got
>> > > >> > picked up and where it is in queue.
>> > > >> > 3. Eliminate network hop from Bigtop box to our PTest master.
>> > > >> >
>> > > >> > The disadvantage is:
>> > > >> > 1. We don't have much experience doing Jenkins admin, but it
>> doesn't
>> > > look
>> > > >> > too bad.  Mostly, restart if there's issue and clean up if out of
>> > > space.
>> > > >> >
>> > > >> > I wonder what people think, and if there's any objections to
>> this?
>> >  If
>> > > >> not,
>> > > >> > I'll try setting up this weekend.  Then, there is some follow-up
>> > work,
>> > > >> like
>> > > >> > changing the Jenkins url's displayed in the test report.
>> > > >> >
>> > > >> > Thanks!
>> > > >> > Szehon
>> > > >> >
>> > > >>
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> > >
>> >
>>
>
>

--f46d04138e5f82151f04f7a9cb47--