Return-Path: X-Original-To: apmail-parquet-commits-archive@minotaur.apache.org Delivered-To: apmail-parquet-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1860218535 for ; Tue, 9 Feb 2016 01:38:00 +0000 (UTC) Received: (qmail 98322 invoked by uid 500); 9 Feb 2016 01:38:00 -0000 Delivered-To: apmail-parquet-commits-archive@parquet.apache.org Received: (qmail 98291 invoked by uid 500); 9 Feb 2016 01:37:59 -0000 Mailing-List: contact commits-help@parquet.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@parquet.apache.org Delivered-To: mailing list commits@parquet.apache.org Received: (qmail 98282 invoked by uid 99); 9 Feb 2016 01:37:59 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Feb 2016 01:37:59 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id CC91BE00B2; Tue, 9 Feb 2016 01:37:59 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: julien@apache.org To: commits@parquet.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: parquet-cpp git commit: PARQUET-500: Add gcov build options and add instructions for generating and uploading code coverage Date: Tue, 9 Feb 2016 01:37:59 +0000 (UTC) Repository: parquet-cpp Updated Branches: refs/heads/master f3dbd5a4e -> fdbe437b1 PARQUET-500: Add gcov build options and add instructions for generating and uploading code coverage Since web-based coverage sites rely on a private token, we won't want to include coverage builds in Travis CI for now, but I figured out the magical incantations to perform a manual coverage run and upload it to either coveralls or codecov. See the results here: https://coveralls.io/builds/4993369 https://codecov.io/github/wesm/parquet-cpp?ref=fbd0a7b949aa1c8648af8335c37916cab9d0438b I much prefer the codecov.io site UX and it feels a lot more responsive. If we can enable apache/parquet-cpp in codecov.io I can periodically run coverage builds from master if that helps, and we can add a badge to the project README. Small TODO here: in PARQUET-447 we'll want to clean up the compiler flags (e.g. optimization needs to be disabled for coverage runs). Author: Wes McKinney Closes #41 from wesm/PARQUET-500 and squashes the following commits: 28f904a [Wes McKinney] Fix typo from Kudu 9607ff9 [Wes McKinney] Edit to be clearer fbd0a7b [Wes McKinney] Add codecov.io instructions a1f7510 [Wes McKinney] Add coverage helper script and add instructions for posting coverage manually to coveralls.io 017a571 [Wes McKinney] Add coverage config from Kudu Project: http://git-wip-us.apache.org/repos/asf/parquet-cpp/repo Commit: http://git-wip-us.apache.org/repos/asf/parquet-cpp/commit/fdbe437b Tree: http://git-wip-us.apache.org/repos/asf/parquet-cpp/tree/fdbe437b Diff: http://git-wip-us.apache.org/repos/asf/parquet-cpp/diff/fdbe437b Branch: refs/heads/master Commit: fdbe437b1b6e85385a41a0038637cc4c87d4d640 Parents: f3dbd5a Author: Wes McKinney Authored: Mon Feb 8 17:37:56 2016 -0800 Committer: Julien Le Dem Committed: Mon Feb 8 17:37:56 2016 -0800 ---------------------------------------------------------------------- CMakeLists.txt | 27 +++++++++++++++++++ README.md | 48 ++++++++++++++++++++++++++++++++++ build-support/collect_coverage.py | 43 ++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/fdbe437b/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/CMakeLists.txt b/CMakeLists.txt index 6d150e2..d9d9a3f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -21,6 +21,8 @@ project(parquet-cpp) if (NOT "$ENV{PARQUET_GCC_ROOT}" STREQUAL "") set(GCC_ROOT $ENV{PARQUET_GCC_ROOT}) set(CMAKE_C_COMPILER ${GCC_ROOT}/bin/gcc) + set(GCOV_PATH ${GCC_ROOT}/bin/gcov) + set(CMAKE_CXX_COMPILER ${GCC_ROOT}/bin/g++) set(CMAKE_CXX_COMPILER ${GCC_ROOT}/bin/g++) endif() @@ -227,6 +229,30 @@ set(PARQUET_MIN_TEST_LIBS parquet) set(PARQUET_TEST_LINK_LIBS ${PARQUET_MIN_TEST_LIBS}) +############################################################# +# Code coverage + +# Adapted from Apache Kudu (incubating) +if ("${PARQUET_GENERATE_COVERAGE}") + if("${CMAKE_CXX_COMPILER}" MATCHES ".*clang.*") + # There appears to be some bugs in clang 3.3 which cause code coverage + # to have link errors, not locating the llvm_gcda_* symbols. + # This should be fixed in llvm 3.4 with http://llvm.org/viewvc/llvm-project?view=revision&revision=184666 + message(SEND_ERROR "Cannot currently generate coverage with clang") + endif() + message(STATUS "Configuring build for gcov") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 --coverage") + # For coverage to work properly, we need to use static linkage. Otherwise, + # __gcov_flush() doesn't properly flush coverage from every module. + # See http://stackoverflow.com/questions/28164543/using-gcov-flush-within-a-library-doesnt-force-the-other-modules-to-yield-gc + if("${PARQUET_LINK}" STREQUAL "a") + message("Using static linking for coverage build") + set(PARQUET_LINK "s") + elseif("${PARQUET_LINK}" STREQUAL "d") + message(SEND_ERROR "Cannot use coverage with dynamic linking") + endif() +endif() + ############################################################ # Library config @@ -286,4 +312,5 @@ add_custom_target(clean-all # installation install(TARGETS parquet + ARCHIVE DESTINATION lib LIBRARY DESTINATION lib) http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/fdbe437b/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index a809713..1578d4e 100644 --- a/README.md +++ b/README.md @@ -114,3 +114,51 @@ For error handling, this project uses exceptions. In general, many of the APIs at the layers are interface based for extensibility. To minimize the cost of virtual calls, the APIs should be batch-centric. For example, encoding should operate on batches of values rather than a single value. + +## Code Coverage + +To build with `gcov` code coverage and upload results to http://coveralls.io or +http://codecov.io, here are some instructions. + +First, build the project with coverage and run the test suite + +``` +cd $PARQUET_HOME +mkdir coverage-build +cd coverage-build +cmake -DPARQUET_GENERATE_COVERAGE=1 +make -j$PARALLEL +ctest +``` + +The `gcov` artifacts are not located in a place that works well with either +coveralls or codecov, so there is a helper script you need to run + +``` +mkdir coverage_artifacts +python ../build-support/collect_coverage.py CMakeFiles/parquet.dir/src/ coverage_artifacts +``` + +For codecov.io (using the provided project token -- be sure to keep this +private): + +``` +cd coverage_artifacts +codecov --token $PARQUET_CPP_CODECOV_TOKEN --gcov-args '\-l' --root $PARQUET_ROOT +``` + +For coveralls, install `cpp_coveralls`: + +``` +pip install cpp_coveralls +``` + +And the coveralls upload script: + +``` +coveralls -t $PARQUET_CPP_COVERAGE_TOKEN --gcov-options '\-l' -r $PARQUET_ROOT --exclude $PARQUET_ROOT/thirdparty --exclude $PARQUET_ROOT/build --exclude $NATIVE_TOOLCHAIN --exclude $PARQUET_ROOT/src/parquet/thrift +``` + + +Note that `gcov` throws off artifacts from the STL, so I excluded my toolchain +root stored in `$NATIVE_TOOLCHAIN` to avoid a cluttered coverage report. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/fdbe437b/build-support/collect_coverage.py ---------------------------------------------------------------------- diff --git a/build-support/collect_coverage.py b/build-support/collect_coverage.py new file mode 100755 index 0000000..6712760 --- /dev/null +++ b/build-support/collect_coverage.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import os +import os.path as osp +import shutil +import sys + + +def is_coverage_file(path): + return path.endswith('gcno') or path.endswith('gcda') + + +def copy_files(path, outpath='.'): + for root, dirs, files in os.walk(path): + for fname in files: + if not is_coverage_file(fname): + continue + relpath = osp.join(root, fname) + dstpath = '_'.join((root.replace(path, '').replace('/', '_'), + fname)) + + shutil.copy(relpath, osp.join(outpath, dstpath)) + +if __name__ == '__main__': + path = sys.argv[1] + outpath = sys.argv[2] + copy_files(path, outpath)