parquet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jul...@apache.org
Subject parquet-cpp git commit: PARQUET-471: Use common build environment setup for sandbox and Travis CI. Travis CI cleanup
Date Thu, 18 Feb 2016 23:29:49 GMT
Repository: parquet-cpp
Updated Branches:
  refs/heads/master 4ae1b921b -> 17a204555


PARQUET-471: Use common build environment setup for sandbox and Travis CI. Travis CI cleanup

This also adds better build instructions for OS X users using Homebrew (googletest is not
available and needs to be build from `thirdparty/`).

Closes #53

Author: Wes McKinney <wesm@apache.org>

Closes #54 from wesm/PARQUET-471 and squashes the following commits:

162f523 [Wes McKinney] Refactor .travis.yml and associated scripts to have a common sandbox
setup script. Add a build matrix and disable unneeded OS X gcc build. Better thirdparty documentation
for googletest and build setup


Project: http://git-wip-us.apache.org/repos/asf/parquet-cpp/repo
Commit: http://git-wip-us.apache.org/repos/asf/parquet-cpp/commit/17a20455
Tree: http://git-wip-us.apache.org/repos/asf/parquet-cpp/tree/17a20455
Diff: http://git-wip-us.apache.org/repos/asf/parquet-cpp/diff/17a20455

Branch: refs/heads/master
Commit: 17a204555bede41d1f68e59a540f3bbef072c821
Parents: 4ae1b92
Author: Wes McKinney <wesm@apache.org>
Authored: Thu Feb 18 15:29:46 2016 -0800
Committer: Julien Le Dem <julien@dremio.com>
Committed: Thu Feb 18 15:29:46 2016 -0800

----------------------------------------------------------------------
 .travis.yml                      | 49 +++++++++++++++++++++--------------
 README.md                        | 45 ++++++++++++++++++++++++++------
 ci/before_script_travis.sh       | 20 ++++----------
 ci/run_tests.sh                  |  5 ----
 setup_build_env.sh               | 22 +++++++---------
 thirdparty/set_thirdparty_env.sh | 18 +++++++++++++
 6 files changed, 98 insertions(+), 61 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index 09f4705..ad71e29 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,15 +1,5 @@
 sudo: required
 dist: trusty
-language: cpp
-
-compiler:
-    - gcc
-    - clang
-
-os:
-    - linux
-    - osx
-
 addons:
   apt:
     sources:
@@ -28,17 +18,36 @@ addons:
     - bison                        #needed for thrift cpp compilation
     - flex                         #needed for thrift cpp compilation
     - pkg-config                   #needed for thrift cpp compilation
+cache:
+  directories:
+  - $HOME/parquet-build
 
+matrix:
+  include:
+  - compiler: gcc
+    os: linux
+    script:
+    - make lint
+    - make -j4 || exit 1
+    - valgrind --tool=memcheck --leak-check=yes --error-exitcode=1 ctest
+  - compiler: clang
+    os: linux
+    script:
+    - make -j4 || exit 1
+    - ctest
+  - os: osx
+    compiler: clang
+    addons:
+    script:
+    - make -j4 || exit 1
+    - ctest
+
+language: cpp
 before_install:
-    - mkdir $HOME/build_dir
-    - cd $HOME/build_dir
+- mkdir $HOME/parquet-build
+- pushd $HOME/parquet-build
 
 before_script:
-    - source $TRAVIS_BUILD_DIR/ci/before_script_travis.sh
-    - cmake -DCMAKE_CXX_FLAGS="-Werror" $TRAVIS_BUILD_DIR
-    - export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data
-
-script:
-- make
-- source $TRAVIS_BUILD_DIR/ci/run_tests.sh
-- make lint
+- source $TRAVIS_BUILD_DIR/ci/before_script_travis.sh
+- cmake -DCMAKE_CXX_FLAGS="-Werror" $TRAVIS_BUILD_DIR
+- export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data

http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 1578d4e..b6ec89e 100644
--- a/README.md
+++ b/README.md
@@ -1,35 +1,64 @@
-Parquet-cpp [![Build Status](https://travis-ci.org/apache/parquet-cpp.svg)](https://travis-ci.org/apache/parquet-cpp)
-===========
-A C++ library to read parquet files.
+## parquet-cpp: a C++ library to read and write the Apache Parquet columnar data format.
+
+[![Build Status](https://travis-ci.org/apache/parquet-cpp.svg)](https://travis-ci.org/apache/parquet-cpp)
 
 ## Third Party Dependencies
+
 - snappy
 - lz4
+- zlib
 - thrift 0.7+ [install instructions](https://thrift.apache.org/docs/install/)
+- googletest 1.7.0 (cannot be installed with package managers)
+
+You can install these dependencies using a package manager or using the
+`thirdparty/` scripts in this repository. On Homebrew, you can run:
 
-Many package managers support some or all of these dependencies. E.g.:
 ```shell
-ubuntu$ sudo apt-get install libboost-dev libsnappy-dev liblz4-dev
+ brew install snappy lz4 thrift zlib
 ```
+
+To build the thirdparty libraries in-tree, run:
+
 ```shell
-mac$ brew install snappy lz4 thrift
+./thirdparty/download_thirdparty.sh
+./thirdparty/build_thirdparty.sh
+source thirdparty/set_thirdparty_env.sh
 ```
 
-`setup_build_env.sh` tries to automate setting up a build environment for you
+The provided script `setup_build_env.sh` sets up a build environment for you
 with third party dependencies.  You use it by running `source
 setup_build_env.sh`.  By default, it will create a build directory `build/`.
 You can override the build directory by setting the BUILD_DIR env variable to
 another location.
 
-Also feel free to take a look at our [.travis.yml](.travis.yml) to see how that build env
is set up.
+After building the thirdparty libraries, for future development iteration you
+can set the dependency environment variables (detailed below) by running
+
+`source $BUILD_DIR/thirdparty/set_thirdparty_env.sh`
+
+Note, the environment variables are set automatically the first time you run
+`setup_build_env.sh`.
 
+The unit tests depend on `googletest` which cannot be installed with Homebrew
+or normal package managers. If you wish to use system dependencies, we
+recommend that you build googletest in-tree by running:
+
+```
+./thirdparty/download_thirdparty.sh
+./thirdparty/build_thirdparty.sh gtest
+source thirdparty/versions.sh
+export GTEST_HOME=`pwd`/thirdparty/$GTEST_BASEDIR
+```
 
 ## Build
+
 - `cmake .`
+
   - You can customize dependent library locations through various environment variables:
     - THRIFT_HOME customizes the thrift installed location.
     - SNAPPY_HOME customizes the snappy installed location.
     - LZ4_HOME customizes the lz4 installed location.
+
 - `make`
 
 The binaries will be built to ./debug which contains the libraries to link against as

http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/ci/before_script_travis.sh
----------------------------------------------------------------------
diff --git a/ci/before_script_travis.sh b/ci/before_script_travis.sh
index b1f31c5..c467c67 100755
--- a/ci/before_script_travis.sh
+++ b/ci/before_script_travis.sh
@@ -3,25 +3,15 @@
 # Build an isolated thirdparty
 cp -r $TRAVIS_BUILD_DIR/thirdparty .
 ./thirdparty/download_thirdparty.sh
-source thirdparty/versions.sh
 
 if [ $TRAVIS_OS_NAME == "osx" ]; then
-  brew update
-  brew install thrift lz4 snappy
-
-  # Only build gtest
-  ./thirdparty/build_thirdparty.sh gtest
-fi
-
-if [ $TRAVIS_OS_NAME == "linux" ]; then
-  ./thirdparty/build_thirdparty.sh
-  export THRIFT_HOME=$HOME/build_dir/thirdparty/installed
-  export SNAPPY_HOME=$HOME/build_dir/thirdparty/installed
-  export LZ4_HOME=$HOME/build_dir/thirdparty/installed
-
+  brew update -q
+  brew install thrift
+else
   # Use a C++11 compiler on Linux
   export CC="gcc-4.9"
   export CXX="g++-4.9"
 fi
 
-export GTEST_HOME=$HOME/build_dir/thirdparty/$GTEST_BASEDIR
+export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data
+source $TRAVIS_BUILD_DIR/setup_build_env.sh

http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/ci/run_tests.sh
----------------------------------------------------------------------
diff --git a/ci/run_tests.sh b/ci/run_tests.sh
deleted file mode 100755
index aa7b2f6..0000000
--- a/ci/run_tests.sh
+++ /dev/null
@@ -1,5 +0,0 @@
-if [ $TRAVIS_OS_NAME == "linux" ]; then
-  valgrind --tool=memcheck --leak-check=yes --error-exitcode=1 ctest;
-else
-  ctest;
-fi

http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/setup_build_env.sh
----------------------------------------------------------------------
diff --git a/setup_build_env.sh b/setup_build_env.sh
index 6df1f49..60ce7cf 100755
--- a/setup_build_env.sh
+++ b/setup_build_env.sh
@@ -3,27 +3,23 @@
 SOURCE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
 : ${BUILD_DIR:=$SOURCE_DIR/build}
 
+# Create an isolated thirdparty
 mkdir -p $BUILD_DIR
 cp -r $SOURCE_DIR/thirdparty $BUILD_DIR
-cd $BUILD_DIR
+pushd $BUILD_DIR
+
 ./thirdparty/download_thirdparty.sh
 ./thirdparty/build_thirdparty.sh
-source thirdparty/versions.sh
-
-export SNAPPY_HOME=$BUILD_DIR/thirdparty/installed
-export LZ4_HOME=$BUILD_DIR/thirdparty/installed
-export ZLIB_HOME=$BUILD_DIR/thirdparty/installed
-# build script doesn't support building thrift on OSX
-if [ "$(uname)" != "Darwin" ]; then
-  export THRIFT_HOME=$BUILD_DIR/thirdparty/installed
-fi
+source thirdparty/set_thirdparty_env.sh
 
 export PARQUET_TEST_DATA=$SOURCE_DIR/data
-export GTEST_HOME=$BUILD_DIR/thirdparty/$GTEST_BASEDIR
 
-cmake $SOURCE_DIR
+# Thrift needs to be installed from brew on OS X
+if [ "$(uname)" == "Darwin" ]; then
+  export -n THRIFT_HOME
+fi
 
-cd $SOURCE_DIR
+popd
 
 echo
 echo "Build env initialized in $BUILD_DIR."

http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/thirdparty/set_thirdparty_env.sh
----------------------------------------------------------------------
diff --git a/thirdparty/set_thirdparty_env.sh b/thirdparty/set_thirdparty_env.sh
new file mode 100644
index 0000000..50c5731
--- /dev/null
+++ b/thirdparty/set_thirdparty_env.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+SOURCE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
+source $SOURCE_DIR/versions.sh
+
+if [ -z "$THIRDPARTY_DIR" ]; then
+	THIRDPARTY_DIR=$SOURCE_DIR
+fi
+
+export SNAPPY_HOME=$THIRDPARTY_DIR/installed
+export LZ4_HOME=$THIRDPARTY_DIR/installed
+export ZLIB_HOME=$THIRDPARTY_DIR/installed
+# build script doesn't support building thrift on OSX
+if [ "$(uname)" != "Darwin" ]; then
+  export THRIFT_HOME=$THIRDPARTY_DIR/installed
+fi
+
+export GTEST_HOME=$THIRDPARTY_DIR/$GTEST_BASEDIR


Mime
View raw message