arrow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From w...@apache.org
Subject arrow git commit: ARROW-905 [Docs] Dockerize document generation
Date Thu, 12 Oct 2017 04:49:29 GMT
Repository: arrow
Updated Branches:
  refs/heads/master 4cb3e97e9 -> 60cb1c372


ARROW-905 [Docs] Dockerize document generation

This PR has two parts. First part is to use Docker to generate the API documentation for C,
C++, Python and Java. This part is now complete.
The second part is to be able to run the site locally, which is still in progress.

Author: Heimir Sverrisson <heimir.sverrisson@gmail.com>

Closes #1162 from heimir-sverrisson/hs/dockerize_api_docs and squashes the following commits:

f758fdc3 [Heimir Sverrisson] ARROW-905 [Docs] Add Dockerfile for reproducible documentation
generation


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/60cb1c37
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/60cb1c37
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/60cb1c37

Branch: refs/heads/master
Commit: 60cb1c372a7c4cf98aeeb5b9deb1ab6c67a5956a
Parents: 4cb3e97
Author: Heimir Sverrisson <heimir.sverrisson@gmail.com>
Authored: Thu Oct 12 00:49:23 2017 -0400
Committer: Wes McKinney <wes.mckinney@twosigma.com>
Committed: Thu Oct 12 00:49:23 2017 -0400

----------------------------------------------------------------------
 c_glib/.gitignore                   |   1 +
 dev/README.md                       |  32 ++++++++-
 dev/docker-compose.yml              |  30 +++++++++
 dev/gen_apidocs.sh                  |  21 ++++++
 dev/gen_apidocs/Dockerfile          |  83 +++++++++++++++++++++++
 dev/gen_apidocs/create_documents.sh | 110 +++++++++++++++++++++++++++++++
 dev/run_docker_compose.sh           |  40 +++++++++++
 dev/run_site.sh                     |  21 ++++++
 dev/run_site/Dockerfile             |  34 ++++++++++
 dev/run_site/run_site.sh            |  24 +++++++
 java/.gitignore                     |   1 +
 site/.gitignore                     |   2 +
 site/_config.yml                    |   3 +
 13 files changed, 401 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/c_glib/.gitignore
----------------------------------------------------------------------
diff --git a/c_glib/.gitignore b/c_glib/.gitignore
index ff3615a..373ae73 100644
--- a/c_glib/.gitignore
+++ b/c_glib/.gitignore
@@ -30,6 +30,7 @@ Makefile.in
 /doc/reference/*.stamp
 /doc/reference/html/
 /doc/reference/xml/
+/doc/reference/tmpl/
 /libtool
 /m4/
 /stamp-h1

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/README.md
----------------------------------------------------------------------
diff --git a/dev/README.md b/dev/README.md
index 91d688d..05a68fc 100644
--- a/dev/README.md
+++ b/dev/README.md
@@ -109,4 +109,34 @@ Studio 2015):
 
 ```
 dev/release/verify-release-candidate.bat apache-arrow-0.7.0.tar.gz
-```
\ No newline at end of file
+```
+
+## Creating API documentation
+
+The generation of API documentation for `C++`, `C Glib`, `Python` 
+and `Java` has been Dockerized. To generate the API documentation
+run the following command:
+
+```shell
+bash dev/gen_apidocs.sh
+```
+
+This script assumes that the `parquet-cpp` Git repository 
+https://github.com/apache/parquet-cpp has been cloned
+besides the Arrow repository and a `dist` directory can be created
+at the same level by the current user. Please note that most of the
+software must be built in order to create the documentation, so this
+step may take some time to run, especially the first time around as the
+Docker container will also have to be built.
+
+After successfully creating the API documentation the website can be
+run locally to browse the API documentation from the top level
+`Documentation` menu. To run the website issue the command:
+
+```shell
+bash dev/run_site.sh
+```
+
+The local URL for the website running inside the docker container
+will be shown as `Server address:` in the output of the command.
+To stop the server press `Ctrl-C` in that window.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/docker-compose.yml
----------------------------------------------------------------------
diff --git a/dev/docker-compose.yml b/dev/docker-compose.yml
new file mode 100644
index 0000000..7bd2cd4
--- /dev/null
+++ b/dev/docker-compose.yml
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: '3'
+services:
+  gen_apidocs:
+    build: 
+      context: gen_apidocs
+    volumes:
+     - ../..:/apache-arrow
+  run_site:
+    build:
+      context: run_site
+    ports:
+    - "4000:4000"
+    volumes:
+     - ../..:/apache-arrow

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/gen_apidocs.sh
----------------------------------------------------------------------
diff --git a/dev/gen_apidocs.sh b/dev/gen_apidocs.sh
new file mode 100755
index 0000000..4bba226
--- /dev/null
+++ b/dev/gen_apidocs.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Pass the service name to run_docker_compse.sh
+# Which validates environment and runs the service
+exec "$(dirname ${BASH_SOURCE})"/run_docker_compose.sh gen_apidocs

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/gen_apidocs/Dockerfile
----------------------------------------------------------------------
diff --git a/dev/gen_apidocs/Dockerfile b/dev/gen_apidocs/Dockerfile
new file mode 100644
index 0000000..0b2844c
--- /dev/null
+++ b/dev/gen_apidocs/Dockerfile
@@ -0,0 +1,83 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+FROM ubuntu:14.04
+ADD . /apache-arrow
+WORKDIR /apache-arrow
+# Prerequsites for apt-add-repository
+RUN apt-get update && apt-get install -y \
+    software-properties-common python-software-properties
+# Basic OS dependencies
+RUN apt-add-repository -y ppa:ubuntu-toolchain-r/test && \
+    apt-get update && apt-get install -y \
+        wget \
+        rsync \
+        git \
+        gcc-4.9 \
+        g++-4.9 \
+        build-essential 
+# This will install conda in /home/ubuntu/miniconda
+RUN wget -O /tmp/miniconda.sh \
+    https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
+    bash /tmp/miniconda.sh -b -p /home/ubuntu/miniconda && \
+    rm /tmp/miniconda.sh
+# C++ dependencies
+RUN /home/ubuntu/miniconda/bin/conda install -c conda-forge \
+    boost-cpp \
+    doxygen \
+    maven \
+    cmake \
+    zlib \
+    thrift-cpp
+# C_Glib dependencies
+RUN apt-get install -y \
+    libgtk2.0-dev \
+    gtk-doc-tools \
+    gobject-introspection \
+    libgirepository1.0-dev \
+    autogen \
+    autoconf-archive
+# Python dependencies
+RUN apt-get install -y \
+    pkg-config
+# Create Conda environment
+RUN /home/ubuntu/miniconda/bin/conda create -y -q -n pyarrow-dev \
+        # Python
+        python=3.6 \
+        numpy \
+        pandas \
+        pytest \
+        cython \
+        ipython \
+        matplotlib \
+        numpydoc \
+        sphinx \
+        sphinx_bootstrap_theme \
+        six \
+        setuptools \
+        # C++
+        cmake \
+        flatbuffers \
+        rapidjson \
+        thrift-cpp \
+        snappy \
+        zlib \
+        brotli \
+        jemalloc \
+        lz4-c \
+        zstd \
+        -c conda-forge
+CMD arrow/dev/gen_apidocs/create_documents.sh

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/gen_apidocs/create_documents.sh
----------------------------------------------------------------------
diff --git a/dev/gen_apidocs/create_documents.sh b/dev/gen_apidocs/create_documents.sh
new file mode 100755
index 0000000..573a328
--- /dev/null
+++ b/dev/gen_apidocs/create_documents.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Set up environment and output directory for C++ libraries
+cd /apache-arrow
+rm -rf dist
+mkdir dist
+export ARROW_BUILD_TYPE=release
+export ARROW_HOME=$(pwd)/dist
+export PARQUET_HOME=$(pwd)/dist
+CONDA_BASE=/home/ubuntu/miniconda
+export LD_LIBRARY_PATH=$(pwd)/dist/lib:${CONDA_BASE}/lib:${LD_LIBRARY_PATH}
+export THRIFT_HOME=${CONDA_BASE}
+export BOOST_ROOT=${CONDA_BASE}
+export PATH=${CONDA_BASE}/bin:${PATH}
+
+# Prepare the asf-site before copying api docs
+pushd arrow/site
+rm -rf asf-site
+export GIT_COMMITTER_NAME="Nobody"
+export GIT_COMMITTER_EMAIL="nobody@nowhere.com"
+git clone --branch=asf-site \
+    https://git-wip-us.apache.org/repos/asf/arrow-site.git asf-site
+popd
+
+# Make Python documentation (Depends on C++ )
+# Build Arrow C++
+source activate pyarrow-dev
+rm -rf arrow/cpp/build
+mkdir arrow/cpp/build
+pushd arrow/cpp/build
+cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
+      -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
+      -DARROW_PYTHON=on \
+      -DARROW_PLASMA=on \
+      -DARROW_BUILD_TESTS=OFF \
+      ..
+make -j4
+make install
+popd
+
+# Build Parquet C++
+rm -rf parquet-cpp/build
+mkdir parquet-cpp/build
+pushd parquet-cpp/build
+cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
+      -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
+      -DPARQUET_BUILD_BENCHMARKS=off \
+      -DPARQUET_BUILD_EXECUTABLES=off \
+      -DPARQUET_BUILD_TESTS=off \
+      ..
+make -j4
+make install
+popd
+
+# Now Python documentation can be built
+pushd arrow/python
+rm -rf build/*
+rm -rf doc/_build
+python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
+    --with-plasma --with-parquet --inplace
+python setup.py build_sphinx -s doc/source
+mkdir -p ../site/asf-site/docs/python
+rsync -r doc/_build/html/ ../site/asf-site/docs/python
+popd
+
+# Build c_glib documentation
+pushd arrow/c_glib
+rm -rf doc/reference/html/*
+./autogen.sh
+./configure \
+    --with-arrow-cpp-build-dir=$(pwd)/../cpp/build \
+    --with-arrow-cpp-build-type=$ARROW_BUILD_TYPE \
+    --enable-gtk-doc
+LD_LIBRARY_PATH=$(pwd)/../cpp/build/$ARROW_BUILD_TYPE make GTK_DOC_V_XREF=": "
+mkdir -p ../site/asf-site/docs/c_glib
+rsync -r doc/reference/html/ ../site/asf-site/docs/c_glib
+popd
+
+# Make C++ documentation
+pushd arrow/cpp/apidoc
+rm -rf html/*
+doxygen Doxyfile
+mkdir -p ../../site/asf-site/docs/cpp
+rsync -r html/ ../../site/asf-site/docs/cpp
+popd
+
+# Make Java documentation
+pushd arrow/java
+rm -rf target/site/apidocs/*
+mvn -Drat.skip=true install
+mvn -Drat.skip=true site
+mkdir -p ../site/asf-site/docs/java/
+rsync -r target/site/apidocs ../site/asf-site/docs/java/
+popd

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_docker_compose.sh
----------------------------------------------------------------------
diff --git a/dev/run_docker_compose.sh b/dev/run_docker_compose.sh
new file mode 100755
index 0000000..f46879e
--- /dev/null
+++ b/dev/run_docker_compose.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ $# -lt 1 ]; then
+    echo "This script takes one argument - the docker service to run" >&2
+    exit 1
+fi
+
+# Make sure this is always run in the directory above Arrow root
+cd $(dirname "${BASH_SOURCE}")/../..
+
+if [ ! -d arrow ]; then
+    echo "Please make sure that the top level Arrow directory" >&2
+    echo "is named 'arrow'"
+    exit 1
+fi
+
+if [ ! -d parquet-cpp ]; then
+    echo "Please clone the Parquet repo next to the Arrow repo" >&2
+    exit 1
+fi
+
+GID=$(id -g ${USERNAME})
+docker-compose -f arrow/dev/docker-compose.yml run \
+               -u "${UID}:${GID}" "${1}"

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_site.sh
----------------------------------------------------------------------
diff --git a/dev/run_site.sh b/dev/run_site.sh
new file mode 100755
index 0000000..1e64dad
--- /dev/null
+++ b/dev/run_site.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Pass the service name to run_docker_compse.sh
+# Which validates environment and runs the service
+exec "$(dirname ${BASH_SOURCE})"/run_docker_compose.sh run_site

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_site/Dockerfile
----------------------------------------------------------------------
diff --git a/dev/run_site/Dockerfile b/dev/run_site/Dockerfile
new file mode 100644
index 0000000..cea4ceb
--- /dev/null
+++ b/dev/run_site/Dockerfile
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM ubuntu:14.04
+ADD . /apache-arrow
+WORKDIR /apache-arrow
+# Prerequsites for apt-add-repository
+RUN apt-get update && apt-get install -y \
+    software-properties-common python-software-properties
+# Set up Ruby repository
+RUN apt-add-repository ppa:brightbox/ruby-ng
+# The publication tools
+RUN apt-get update; apt-get install -y \
+    apt-transport-https \
+    ruby2.2-dev \
+    ruby2.2 \
+    zlib1g-dev \ 
+    make \
+    gcc
+RUN gem install jekyll bundler
+CMD arrow/dev/run_site/run_site.sh

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_site/run_site.sh
----------------------------------------------------------------------
diff --git a/dev/run_site/run_site.sh b/dev/run_site/run_site.sh
new file mode 100755
index 0000000..8527b2b
--- /dev/null
+++ b/dev/run_site/run_site.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+cd /apache-arrow/arrow/site
+export GEM_HOME=$(pwd)
+export BUNDLE_PATH=$(pwd)
+export HOME=$(pwd)
+bundle install --path .
+HOST_IP=$(grep $(hostname) /etc/hosts | cut -f 1)
+bundle exec jekyll serve -c _config.yml -s asf-site --host ${HOST_IP} --port 4000

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/java/.gitignore
----------------------------------------------------------------------
diff --git a/java/.gitignore b/java/.gitignore
index 73c1be4..03f5bf7 100644
--- a/java/.gitignore
+++ b/java/.gitignore
@@ -20,3 +20,4 @@ CMakeFiles
 Makefile
 cmake_install.cmake
 install_manifest.txt
+?/

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/site/.gitignore
----------------------------------------------------------------------
diff --git a/site/.gitignore b/site/.gitignore
index 46bc466..d3c112a 100644
--- a/site/.gitignore
+++ b/site/.gitignore
@@ -4,3 +4,5 @@ _site
 Gemfile.lock
 asf-site
 build/
+.bundle/
+ruby/

http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/site/_config.yml
----------------------------------------------------------------------
diff --git a/site/_config.yml b/site/_config.yml
index fcb76a3..a6c5575 100644
--- a/site/_config.yml
+++ b/site/_config.yml
@@ -23,6 +23,9 @@ kramdown:
   input: GFM
   syntax_highlighter: rouge
 
+include:
+  - _static
+
 exclude:
   - Gemfile
   - Gemfile.lock


Mime
View raw message