impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4593,IMPALA-4635: fix some python build issues
Date Mon, 06 Mar 2017 22:48:46 GMT
Hello Impala Public Jenkins, Matthew Jacobs, Alex Behm,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/6218

to look at the new patch set (#5).

Change subject: IMPALA-4593,IMPALA-4635: fix some python build issues
......................................................................

IMPALA-4593,IMPALA-4635: fix some python build issues

Build C/C++ packages with toolchain GCC to avoid ABI compatibility
issues. This requires a multi-step bootstrapping process:
1. install basic non-C/C++ packages into the virtualenv
2. use Python 2.7 from the virtualenv to bootstrap the toolchain
3. use toolchain gcc to build C/C++ packages
4. build the kudu-python package with toolchain gcc and Cython

To avoid potentially pulling in cached versions of packages
built with a different compiler, this patch also disables pip's
caching. This should not have a significant effect on performance
since we've enabled ccache and cache downloaded packages in
infra/python/deps.

Improve bootstrapping time significantly by using ccache and by
parallelising the numpy build - the most expensive part of the
install process. On a system with a warmed-up ccache,
bootstrapping after deleting infra/python/env takes 1m16s. Previously
it could take over 5m.

Testing:
Tested manually on Ubuntu 16.04 to confirm that it fixes the ABI
problem mentioned in IMPALA-4593. Initially "import kudu" failed
in my dev environment. After deleting infra/python/env and
re-bootstrapping, "import kudu" succeeded.

Also ran the standard test suite on CentOS 6 and built Impala on
a range of platforms (CentOS 5,6,7; SLES 11,12; Debian 6,7;
Ubuntu12.04,14.04,16.04) to make sure nothing broke.

Change-Id: I9e807510eddeb354069e0478363f649a1c1b75cf
---
M infra/python/README
M infra/python/bootstrap_virtualenv.py
A infra/python/deps/compiled-requirements.txt
M infra/python/deps/download_requirements
A infra/python/deps/kudu-requirements.txt
M infra/python/deps/pip_download.py
M infra/python/deps/requirements.txt
7 files changed, 197 insertions(+), 96 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/6218/5
-- 
To view, visit http://gerrit.cloudera.org:8080/6218
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9e807510eddeb354069e0478363f649a1c1b75cf
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>

Mime
View raw message