Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAKm0gCSeYy_ZzVcr+iOzyx4zfxmoF304nOUjtsHMNdTymQAFmA@mail.gmail.com>
References: <CACJVYb7pPo6uFD-oKvEsi6B6g=ioE6BdE-aa2dRGdadj6UiLbg@mail.gmail.com>
 <CAPcia-rQh4kGkV0wThqsOabUnFYX8GkVbFMfAQDZ_J+aOeT5Zg@mail.gmail.com>
 <CAC-pSX0VxM6cdC2LWjdL8jAGuNm5qiMq05-YyRSqtrsO1WBFpw@mail.gmail.com>
 <CAPcia-o2ZRTdB4J4N7yUPX4tbrCe7VnXVhHdcA2i_X3qffQWnA@mail.gmail.com>
 <CACJVYb5VzM=Bb-xAaxMy3BgVxDL0XMbJY5hbNRm9QO+hbqXj4g@mail.gmail.com>
 <CAPcia-qYq3JkPn2XkKUu0H1Kek2Txv-Gdjmz1r=c6VvZNEFLLw@mail.gmail.com>
 <CACJVYb7KPo8gaebb0ppKN6Fgr06Pc+fyYzrr4jDH8oUH5q-H9A@mail.gmail.com>
 <CAC-pSX0WzwLZVqKEPiTz41qUYQ1wicSuQftfM5Jqqfp3jszc=g@mail.gmail.com> <CAKm0gCSeYy_ZzVcr+iOzyx4zfxmoF304nOUjtsHMNdTymQAFmA@mail.gmail.com>
From: Sailesh Mukil <sailesh@cloudera.com>
Date: Thu, 25 May 2017 10:06:13 -0700
Message-ID: <CACJVYb5+fkO0zjt9+oat2k2kNObuHHxvg7gjrENxsK4aA-R_oQ@mail.gmail.com>
Subject: Re: GVO machines dependencies
To: dev@impala.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a114b381261da2305505c3fb9"
archived-at: Thu, 25 May 2017 17:07:01 -0000

--001a114b381261da2305505c3fb9
Content-Type: text/plain; charset="UTF-8"

Glad I could help! The fix is merged now:
https://github.com/awleblang/impala-setup/commit/1fcff44887de76fdb5f86948635c073d0e615f1c

On Thu, May 25, 2017 at 10:00 AM, Laszlo Gaal <laszlo.gaal@cloudera.com>
wrote:

> Thanks a lot for fixing this, Sailesh, this has been bugging me for a while
> when spinning up cloud-based workstations.
>
> On Thu, May 25, 2017 at 6:18 PM, Jim Apple <jbapple@cloudera.com> wrote:
>
> > Nice find, Sailesh!
> >
> > On Wed, May 24, 2017 at 6:39 PM, Sailesh Mukil <sailesh@cloudera.com>
> > wrote:
> > > The issue was that there was a bug in the install.sh script from the
> > > impala-setup/ repo that always assumed that the repo would reside in
> > > /home/$user.
> > >
> > > The Jenkins AMIs seem to have had a very old checkout (8 months old) at
> > > /home/ubuntu/ which was being used for the GVOs. The Jenkins job would
> > make
> > > a fresh checkout in /tmp/ and run install.sh from that checkout. But
> due
> > to
> > > the above bug, the script would change its sources to the old repo in
> > > /home/ubuntu. So, the new dependencies added to the project didn't get
> > > picked up. I've submitted a fix, and will redo the GVO once it's merged
> > > into impala-setup.
> > >
> > > Thanks for your help Michael and Jim!
> > >
> > > On Wed, May 24, 2017 at 4:58 PM, Michael Brown <mikeb@cloudera.com>
> > wrote:
> > >
> > >> "from-scratch" is a misnomer: only the first build to run on a given
> > node
> > >> is truly from-scratch, because running impala-setup takes effect
> > >> system-wide, and is run once, when the node is created.
> > >>
> > >> For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/
> ,
> > >> the
> > >> worker's build history suggests it has been up roughly 17 hours.
> > >> http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20(
> > >> i-0d76efbc8de26926c)/builds
> > >> This means your recent change hasn't taken effect on "older" workers.
> > >>
> > >> Any new workers that get created should get the change.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil <sailesh@cloudera.com>
> > >> wrote:
> > >>
> > >> > Thanks Michael and Jim,
> > >> >
> > >> > It looks like adding to bin/bootstrap_build.sh makes the
> > ub14-build-only
> > >> > job work fine, however, updating the impala-setup still doesn't seem
> > to
> > >> > work. I submitted a pull request that was merged by Dimitris here:
> > >> > https://github.com/awleblang/impala-setup/commits/master
> > >> >
> > >> > Also, the impala-setup chef logs don't seem to get printed in the
> > jenkins
> > >> > console output. So I'm not sure if it's actually being run or not.
> > >> >
> > >> > On Wed, May 24, 2017 at 3:37 PM, Michael Brown <mikeb@cloudera.com>
> > >> wrote:
> > >> >
> > >> > > Thanks Jim; I had forgotten bin/bootstrap_build.sh.
> > >> > >
> > >> > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple <jbapple@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > I'd recommend against #2 using the node setup to install Impala
> > >> > > > dependencies. The only reason it installs openjdk is that
> Jenkins
> > >> > > > needs Java to talk to it. All of the other Impala dependencies
> are
> > >> > > > installed in the jobs themselves.
> > >> > > >
> > >> > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown <
> > mikeb@cloudera.com>
> > >> > > wrote:
> > >> > > > >> Is there a way to add a new dependency to these machines?
> > >> > > > >
> > >> > > > > Our builds use two worker labels:
> > >> > > > > 1. ubuntu14.04-c4.4xlarge-gp2
> > >> > > > > 2. ub14-build-only
> > >> > > > >
> > >> > > > > 1 uses https://github.com/awleblang/impala-setup, and I think
> > you
> > >> > > should
> > >> > > > > add libffi to there. It's something you should do anyway so
> new
> > >> users
> > >> > > get
> > >> > > > > set up with the right dependencies.
> > >> > > > >
> > >> > > > > 2 is a different story, but I see that at
> > >> > > > > http://jenkins.impala.io:8080/configure that worker is
> already
> > >> > > > installing a
> > >> > > > > JDK via apt-get. You could use a similar pattern to install
> > libffi.
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>

--001a114b381261da2305505c3fb9--