flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From m..@apache.org
Subject flink git commit: [FLINK-3887] improve dependency management for building docs
Date Tue, 31 May 2016 09:39:07 GMT
Repository: flink
Updated Branches:
  refs/heads/release-1.0 9ec1fe0a8 -> b033d2a54


[FLINK-3887] improve dependency management for building docs

The Flink documentation build process is currently quite messy. These
changes move us to a new build process with proper dependency
handling. It assures that we all use the same dependency versions for
consistent build output. Also, it eases the automated building process
on other systems (like the ASF Buildbot). The goal was to make the
documentation build process easier and self-contained.

- use Ruby's Bundler Gem to install dependencies
- update README
- adapt Dockerfile
- add additional rules to .gitignore
- change default doc output path from /target to /content
(default path of the flink-web repository)

This closes #2033


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/b033d2a5
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/b033d2a5
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/b033d2a5

Branch: refs/heads/release-1.0
Commit: b033d2a54c25a8be4ff0b0787dbcb1b4c78f25c4
Parents: 9ec1fe0
Author: Maximilian Michels <mxm@apache.org>
Authored: Mon May 16 15:02:59 2016 +0200
Committer: Maximilian Michels <mxm@apache.org>
Committed: Tue May 31 11:38:57 2016 +0200

----------------------------------------------------------------------
 .gitignore             |  7 +++++--
 docs/Gemfile           | 25 +++++++++++++++++++++++++
 docs/README.md         | 30 +++++++++++++++---------------
 docs/build_docs.sh     | 42 ++++++++++++++++++------------------------
 docs/docker/Dockerfile |  6 +-----
 pom.xml                |  5 +++--
 6 files changed, 67 insertions(+), 48 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/b033d2a5/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index 629d62c..9985ffb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,8 +16,6 @@ tmp
 *.jar
 *.log
 .DS_Store
-_site
-docs/api
 build-target
 flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/
 flink-runtime-web/web-dashboard/assets/fonts/
@@ -25,3 +23,8 @@ flink-runtime-web/web-dashboard/node_modules/
 flink-runtime-web/web-dashboard/bower_components/
 atlassian-ide-plugin.xml
 out/
+/docs/api
+/docs/content
+/docs/Gemfile.lock
+/docs/.bundle
+/docs/.rubydeps

http://git-wip-us.apache.org/repos/asf/flink/blob/b033d2a5/docs/Gemfile
----------------------------------------------------------------------
diff --git a/docs/Gemfile b/docs/Gemfile
new file mode 100644
index 0000000..d367999
--- /dev/null
+++ b/docs/Gemfile
@@ -0,0 +1,25 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+source 'https://rubygems.org'
+
+# Dependencies required to build the Flink docs
+gem 'jekyll', '2.5.3'
+gem 'kramdown', '1.10.0'
+gem 'pygments.rb', '0.6.3'
+gem 'therubyracer', '0.12.2'

http://git-wip-us.apache.org/repos/asf/flink/blob/b033d2a5/docs/README.md
----------------------------------------------------------------------
diff --git a/docs/README.md b/docs/README.md
index d37dc77..52dfad3 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -6,19 +6,17 @@ http://flink.apache.org/ is also generated from the files found here.
 
 # Requirements
 
-We use Markdown to write and Jekyll to translate the documentation to static HTML. Kramdown
is
-needed for Markdown processing and the Python based Pygments is used for syntax highlighting.
To run
-Javascript code from Ruby, you need to install a javascript runtime (e.g. `therubyracer`).
You can
-install all needed software via the following commands:
+The dependencies are declared in the Gemfile in this directory. We use Markdown
+to write and Jekyll to translate the documentation to static HTML. All required
+dependencies are installed locally when you build the documentation through the
+`build_docs.sh` script. If you want to install the software manually, use Ruby's
+Bundler Gem to install all dependencies:
 
-    gem install jekyll -v 2.5.3
-    gem install kramdown -v 1.9.0
-    gem install pygments.rb -v 0.6.3
-    gem install therubyracer -v 0.12.2
-    sudo easy_install Pygments
+    gem install bundler
+    bundle install
 
-Note that in Ubuntu based systems, it may be necessary to install the `ruby-dev` and
-`python-setuptools` packages via apt.
+Note that in Ubuntu based systems, it may be necessary to install the `ruby-dev`
+via apt to build native code.
 
 # Using Dockerized Jekyll
 
@@ -35,11 +33,13 @@ The run.sh command brings you in a bash session where you can run following
doc
 
 # Build
 
-The `docs/build_docs.sh` script calls Jekyll and generates the documentation in `docs/target`.
You
-can then point your browser to `docs/target/index.html` and start reading.
+The `docs/build_docs.sh` script installs dependencies locally, calls Jekyll, and
+generates the documentation in `docs/content`. You can then point your browser
+to `docs/content/index.html` and start reading.
 
-If you call the script with the preview flag `build_docs.sh -p`, Jekyll will start a web
server at
-`localhost:4000` and watch the docs directory for updates. Use this mode to preview changes
locally.
+If you call the script with the preview flag `build_docs.sh -p`, Jekyll will
+start a web server at `localhost:4000` and watch the docs directory for
+updates. Use this mode to preview changes locally.
 
 # Contribute
 

http://git-wip-us.apache.org/repos/asf/flink/blob/b033d2a5/docs/build_docs.sh
----------------------------------------------------------------------
diff --git a/docs/build_docs.sh b/docs/build_docs.sh
index cf71b13..412008a 100755
--- a/docs/build_docs.sh
+++ b/docs/build_docs.sh
@@ -17,35 +17,30 @@
 # limitations under the License.
 ################################################################################
 
+set -e
+cd "$(dirname ${BASH_SOURCE[0]})"
 
-HAS_JEKYLL=true
+DIR="`pwd`"
 
-command -v jekyll > /dev/null
-if [ $? -ne 0 ]; then
-	echo -n "ERROR: Could not find jekyll. "
-	echo "Please install with 'gem install jekyll' (see http://jekyllrb.com)."
+# We need at least bundler to proceed
+if [ "`command -v bundle`" == "" ]; then
+	echo "WARN: Could not find bundle."
+    echo "Attempting to install locally. If this doesn't work, please install with 'gem install
bundler'."
 
-	HAS_JEKYLL=false
-fi
-
-command -v redcarpet > /dev/null
-if [ $? -ne 0 ]; then
-	echo -n "WARN: Could not find redcarpet. "
-	echo -n "Please install with 'sudo gem install redcarpet' (see https://github.com/vmg/redcarpet).
"
-	echo "Redcarpet is needed for Markdown parsing and table of contents generation."
-fi
+    # Adjust the PATH to discover the locally installed Ruby gem
+    if which ruby >/dev/null && which gem >/dev/null; then
+        export PATH="$(ruby -rubygems -e 'puts Gem.user_dir')/bin:$PATH"
+    fi
 
-command -v pygmentize > /dev/null
-if [ $? -ne 0 ]; then
-	echo -n "WARN: Could not find pygments. "
-	echo -n "Please install with 'sudo easy_install Pygments' (requires Python; see http://pygments.org).
"
-	echo "Pygments is needed for syntax highlighting of the code examples."
+    # install bundler locally
+    gem install --user-install bundler
 fi
 
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+# Install Ruby dependencies locally
+bundle install --path .rubydeps
 
 DOCS_SRC=${DIR}
-DOCS_DST=${DOCS_SRC}/target
+DOCS_DST=${DOCS_SRC}/content
 
 # default jekyll command is to just build site
 JEKYLL_CMD="build"
@@ -59,6 +54,5 @@ while getopts ":p" opt; do
 	esac
 done
 
-if $HAS_JEKYLL; then
-	jekyll ${JEKYLL_CMD} --source ${DOCS_SRC} --destination ${DOCS_DST}
-fi
+# use 'bundle exec' to insert the local Ruby dependencies
+bundle exec "jekyll ${JEKYLL_CMD} --source '${DOCS_SRC}' --destination '${DOCS_DST}'"

http://git-wip-us.apache.org/repos/asf/flink/blob/b033d2a5/docs/docker/Dockerfile
----------------------------------------------------------------------
diff --git a/docs/docker/Dockerfile b/docs/docker/Dockerfile
index 15d6260..ad4bb30 100644
--- a/docs/docker/Dockerfile
+++ b/docs/docker/Dockerfile
@@ -17,9 +17,5 @@
 FROM centos:centos7
 
 RUN yum install -y vim gem ruby-devel make gcc gcc-c++ python-setuptools && \
-	gem install jekyll -v 2.5.3 && \
-	gem install kramdown -v 1.9.0 && \
-	gem install pygments.rb -v 0.6.3 && \
-	gem install therubyracer -v 0.12.2 && \
-	easy_install Pygments
+    gem install bundler
 

http://git-wip-us.apache.org/repos/asf/flink/blob/b033d2a5/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c62d177..670dbb2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -847,11 +847,12 @@ under the License.
 					</licenseFamilies>
 					<excludes>
 						<!-- Additional files like .gitignore etc.-->
-						<exclude>**/.*</exclude>
+						<exclude>**/.*/**</exclude>
 						<exclude>**/*.prefs</exclude>
 						<exclude>**/*.log</exclude>
 						<!-- External web libraries. -->
 						<exclude>docs/**/bootstrap*</exclude>
+						<exclude>docs/Gemfile.lock</exclude>
 						<exclude>docs/img/*.svg</exclude>
 						<exclude>**/resources/**/font-awesome/**</exclude>
 						<exclude>**/resources/**/jquery*</exclude>
@@ -903,7 +904,7 @@ under the License.
 						<!-- Generated content -->
 						<exclude>out/**</exclude>
 						<exclude>**/target/**</exclude>
-						<exclude>docs/_site/**</exclude>
+						<exclude>docs/content/**</exclude>
 						<exclude>**/scalastyle-output.xml</exclude>
 						<exclude>build-target/**</exclude>
 						<!-- Tools: watchdog -->


Mime
View raw message