Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
Date: Tue, 16 Feb 2016 10:52:13 +0300
From: Konstantin Boudnik <cos@apache.org>
To: dev@hawq.incubator.apache.org
Subject: Re: Dependencies [Was: Build environment]
Message-ID: <20160216075213.GB15300@tpx>
Mail-Followup-To: dev@hawq.incubator.apache.org
References: <20160212065140.GA10978@tpx>
 <CAE44UQe1xgcVOC76T_mgVbgGbR=Lx=XUBPVw18ZK4iZ3euCH+g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="bCsyhTFzCvuiizWE"
Content-Disposition: inline
In-Reply-To: 
 <CAE44UQe1xgcVOC76T_mgVbgGbR=Lx=XUBPVw18ZK4iZ3euCH+g@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)

--bCsyhTFzCvuiizWE
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 16, 2016 at 12:44PM, Lei Chang wrote:
> From simplifying the build & install process standpoint, looks a good idea
> to include libhdfs3 just like libyarn.

Indeed. A quick look further into the code shows, that you guys mightn't ne=
ed
thrift-devel libraries after all, because native thrift APIs aren't used
anywhere (kudos to Roman's findings).

It'd be great to have these issues fixed before the release time. Here's wh=
y:
making an official release to have mandatory _source_ dependencies developed
and hosted elsewhere doesn't sound reasonable nor safe. So, let's please
address it before RC5 hits the voting floor.

Thanks!
  Cos


> On Fri, Feb 12, 2016 at 2:51 PM, Konstantin Boudnik <cos@apache.org> wrot=
e:
>=20
> > Going through this futher. Two dependencies of HAWQ in particular seem =
to
> > be a
> > big snag for the downstream integration. I managed to get most of the b=
its
> > and
> > pieces into tight and compact toolchain, so that the project can be bui=
ld
> > on a
> > generic centos, but there at least two libs that don't fit in to the
> > picture.
> >  - libhdfs3
> >  - thrift-devel
> >
> > From Apache Bigtop standpoint, build dependencies should be either
> > available
> > from the standard distro's repos; or be built from source during the
> > component's package creation. Say, libyarn could be build this way. How=
ever
> > for the two above, either the packages not available from distros or the
> > sources not provided as a part of the project dependencies (again, liby=
arn
> > might be considered as an example). Hence, the question to to the
> > development
> > community here: why libyarn is specifically included as a source
> > dependency,
> > but the other two aren't? Is it possible to add those to the depends/ a=
nd
> > improve the build to the point where they got build before HAWQ, so the
> > autoconf requirements get satisfied?
> >
> > From the UX standpoint of view, it is a very iffi practice to have a so=
urce
> > tree, which has non-standard deps, that have to be downloaded and build
> > separately. That is as a different software projects. Back in early 90s=
 I
> > used
> > to build software for BSD or Solaris, where you had to
> > download,build,install
> > 20+ different libs first, and only then you'd be able to finally start
> > building what you really need. Well, it is 2016 and the Unix world is v=
ery
> > different now. And way more developers' friendly. I'd suggest we borrow
> > from
> > today's practices instead of the quarter-century old ones.
> >
> > libhdfs3 and libyarn are also seem to be the runtime dependencies for t=
he
> > HAWQ, right? Which means, that if hawq is installed as a standard packa=
ge,
> > then those two (and perhaps thrift) will have to be provided as package=
s as
> > well. Which makes the reliance on unknown 3rd party repos a big security
> > no-no.
> >
> > So, any thoughts from the community on the steps to make this better be=
fore
> > the release is out? If we're to add more source code into the project -
> > this
> > is the perfect time to do it.
> >
> > Cos
> >
> > On Thu, Feb 11, 2016 at 12:16PM, Konstantin Boudnik wrote:
> > > Looking a bit further into build dependencies it seems that the
> > environment
> > > relies very heavily on some 3rd party, maintained by someone and
> > somewhere
> > > repos and libs. While it is up to the community on how they handle th=
eir
> > > builds, adding repos and packages which aren't maintained in an open
> > fashion,
> > > won't be helping to attract new contributors. Cases in point
> > >
> > >   - https://bintray.com/wangzw/rpm/rpm
> > >   - http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
> > >   -
> > http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychec=
ker-0.8.19.tar.gz/download
> > >
> > > This certainly will be an integration blocker for the downstream
> > projects like
> > >     https://issues.apache.org/jira/browse/BIGTOP-2323
> > >
> > > One other point (which seems a bit weird). What's the point of
> > >     yum install -y postgresql-devel
> > > followed by
> > >     yum erase -y postgresql postgresql-libs postgresql-devel
> > >
> > > Thanks
> > >   Cos
> > >
> > > On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
> > > > Gents,
> > > >
> > > > have you considered creating and checking-in a wrapper script
> > (run-build.sh or
> > > > whatever) for the build, instead of writing lengthy shell-scripts in
> > Jenkins?
> > > > Then instead of
> > > >
> > > > docker run --rm=3Dtrue -v `pwd`:/data -u root rlei/mydocker:latest
> > /bin/sh -c
> > > > "date; \
> > > >
> > > > cd /data....
> > > >
> > > > [another 23 lines of shell script one has to type each time.....]
> > > >
> > > > one would need to run
> > > >
> > > > docker run --rm=3Dtrue -v `pwd`:/data -w /data -u root
> > rlei/mydocker:latest run-build.sh
> > > >
> > > > and be done with it.
> > > >
> > > > Cos
> > > >
> > > >
> > >
> > >
> >
> >
> >

--bCsyhTFzCvuiizWE
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iF4EAREIAAYFAlbC1S0ACgkQenyFlstYjhKZ8gD+Ow9S4Cmb/dvnEleKLdGzKGlm
TnlMoj9ZDbhfvooMM8MBAMtECKAY6CsBnyF0DT+uJkRiQOpMP7Nt7htfacaD9Yhv
=ZwlO
-----END PGP SIGNATURE-----

--bCsyhTFzCvuiizWE--