kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abu...@apache.org
Subject [kudu] branch master updated: Upgrade docs/site building tools
Date Thu, 27 Aug 2020 10:38:27 GMT
This is an automated email from the ASF dual-hosted git repository.

abukor pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git


The following commit(s) were added to refs/heads/master by this push:
     new de31237  Upgrade docs/site building tools
de31237 is described below

commit de312377d2fa71a4a39f7f9b8cf78615e8d47715
Author: Attila Bukor <abukor@apache.org>
AuthorDate: Wed Aug 26 13:57:28 2020 +0200

    Upgrade docs/site building tools
    
    The tooling to build the documentation for the Kudu website are very
    old making it difficult to run it on newer OSes.
    
    This patch upgrades the Ruby dependencies along with the required
    Doxygen version to make it easier. As the new dependencies are fairly
    new, they require Ruby and Doxygen versions newer than what's included
    by default on CentOS 7 and likely on other OSes that we might want to
    use to build the site. It's fairly simple to upgrade them on the other
    hand, this patch also adds instructions to README.adoc.
    
    Change-Id: I5eddf689da90123ca52075d09de07981d11f8ffe
    Reviewed-on: http://gerrit.cloudera.org:8080/16371
    Tested-by: Kudu Jenkins
    Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
---
 CMakeLists.txt                          |   8 +--
 README.adoc                             |  22 +++++++
 RELEASING.adoc                          |   9 ++-
 docs/support/doxygen/client_api.doxy.in |   4 ++
 docs/support/scripts/Gemfile            |   5 --
 docs/support/scripts/Gemfile.lock       | 101 +++++++++++++++++---------------
 docs/support/scripts/make_docs.sh       |  30 +++++++---
 7 files changed, 112 insertions(+), 67 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index b34c8b2..4b5dcbd 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1369,13 +1369,13 @@ endif (UNIX)
 
 ############################################################
 # "make doxygen" target
-# Requires doxygen of version >= 1.8.11 to leverage the
-# WARN_AS_ERROR configuration parameter (introduced in
-# version 1.8.11) for spotting various issues in the Kudu
+# Requires doxygen of version >= 1.8.19 that includes the
+# fix of a bug that would otherwise fail the build as we
+# treat warnings as errors to spot various issues in the
 # C++ client API documentation.
 ############################################################
 if (UNIX)
-  find_package(Doxygen 1.8.11)
+  find_package(Doxygen 1.8.19)
   if (NOT (DOXYGEN_FOUND AND DOXYGEN_DOT_FOUND))
     message(WARNING "Doxygen with Dot support (graphviz) not found: 'doxygen' target is not
available")
   else ()
diff --git a/README.adoc b/README.adoc
index 6a546da..4603a9f 100644
--- a/README.adoc
+++ b/README.adoc
@@ -279,6 +279,7 @@ $ ./build-support/iwyu.py
 
 See the output of `iwyu.py --help` for details on various modes of operation.
 
+[[building-docs]]
 === Building Kudu documentation
 
 Kudu's documentation is written in asciidoc and lives in the _docs_ subdirectory.
@@ -297,6 +298,17 @@ emitted to _build/docs_. This script requires `ruby` and `gem` to be
installed
 on the system path, and will attempt to install `asciidoctor` and other related
 dependencies into `$HOME/.gems` using https://bundler.io/[bundler].
 
+Some of the dependencies require a recent version of Ruby. To build the
+documentation on a system that comes with an older Ruby version (such as Ruby
+2.0 on CentOS 7), it is easiest to use https://github.com/rbenv/rbenv[rbenv] to
+install Ruby 2.7.
+
+WARNING: As the default values for some configuration options differ between Mac
+and Linux (e.g. file vs log block manager) and the configuration reference is
+generated by running the binaries with `-help`, the documentation *MUST NOT* be
+generated on Mac for publishing purposes, only for verification.
+
+[[updating-the-site]]
 === Updating the Kudu web site documentation
 
 To update the documentation that is integrated into the Kudu web site,
@@ -315,6 +327,13 @@ These will be placed inside the checked-out web site, along with a tarball
 containing only the generated documentation (the _docs/_ and _apidocs/_ paths
 on the web site). Everything can be found in the _build/site_ subdirectory.
 
+To build the C++ Client API you need to have Doxygen 1.8.19 or later which is
+fairly new so you might need to
+https://www.doxygen.nl/manual/install.html#install_src_unix[build it from
+source]. To build it on RHEL/CentOS you'll also need
+https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/[devtoolset] as
+Doxygen uses {cpp}14 since 1.8.17.
+
 You can proceed to commit the changes in the pages repository and send a code
 review for your changes. In the future, this step may be automated whenever
 changes are checked into the main Kudu repository.
@@ -322,6 +341,9 @@ changes are checked into the main Kudu repository.
 After making changes to the `gh-pages` branch, follow the instructions below
 when you want to deploy those changes to the live web site.
 
+WARNING: The site *MUST NOT* be built on Mac for publishing purposes, only for
+verification. See the warning in <<building-docs>> for details.
+
 === Deploying changes to the Apache Kudu web site
 
 When the documentation is updated on the `gh-pages` branch, or when other web
diff --git a/RELEASING.adoc b/RELEASING.adoc
index e0e3f1f..9bd82ad 100644
--- a/RELEASING.adoc
+++ b/RELEASING.adoc
@@ -332,8 +332,13 @@ Note: You can upload to the test PyPi by adding
 . Add the previous release notes to the `prior_release_notes.adoc` and submit the changes.
 
 . Generate the version-specific documentation from that branch following these
-  link:https://github.com/apache/kudu/#updating-the-kudu-web-site-documentation[instructions].
+  <<README.adoc#updating-the-site,instructions>>.
++
+
+WARNING: The site *MUST NOT* be built on Mac. See the
+<<README.adoc#building-docs,documentation building instructions>> for details.
 
++
 . Update the `index.md` file in the `releases` directory. Add a line about
   the newly released version into the `Latest release` section, move line
   about the previous one into the `Previous releases`. Overall, update
@@ -368,7 +373,7 @@ Note: You can upload to the test PyPi by adding
 . Submit these changes to the `gh-pages` Gerrit branch and get them reviewed.
 
 . Once the review is finished and the commit is pushed, update the website following
-  these link:https://github.com/apache/kudu/#deploying-changes-to-the-apache-kudu-web-site[instructions].
+  these <<README.adoc#deploying-changes-to-the-apache-kudu-web-site,instructions>>.
 
 . Add a release blog post about the new release, similar to
   link:https://kudu.apache.org/2019/07/09/apache-kudu-1-10-0-release.html[this].
diff --git a/docs/support/doxygen/client_api.doxy.in b/docs/support/doxygen/client_api.doxy.in
index f7e862c..c438bf5 100644
--- a/docs/support/doxygen/client_api.doxy.in
+++ b/docs/support/doxygen/client_api.doxy.in
@@ -67,3 +67,7 @@ GENERATE_LATEX         = NO
 
 # Do not output warnings on undocumented friend classes/structs/unions.
 HIDE_FRIEND_COMPOUNDS  = YES
+
+# Copy the first member's documentation to the rest in the group if they aren't
+# documented.
+DISTRIBUTE_GROUP_DOC   = YES
diff --git a/docs/support/scripts/Gemfile b/docs/support/scripts/Gemfile
index 0ac227b..e1ffd74 100644
--- a/docs/support/scripts/Gemfile
+++ b/docs/support/scripts/Gemfile
@@ -15,20 +15,15 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# We need to use versions that can run on CentOS 7, which ships Ruby 2.0.0.
-ruby '>= 2.0.0'
-
 source 'https://rubygems.org'
 
 gem 'thread_safe'
 gem 'redcarpet'
 gem 'jekyll'
-gem 'therubyracer' # V8 runtime installer for Jekyll.
 gem 'ref'
 gem 'json'
 gem 'rdoc'
 gem 'yajl-ruby', '>= 1.3.1' # Due to CVE-2017-16516.
-gem 'libv8', '>= 3.16.14.14' # Needed for macOS.
 
 gem 'asciidoctor'
 gem 'tilt'
diff --git a/docs/support/scripts/Gemfile.lock b/docs/support/scripts/Gemfile.lock
index f0587e3..2e1e038 100644
--- a/docs/support/scripts/Gemfile.lock
+++ b/docs/support/scripts/Gemfile.lock
@@ -1,57 +1,69 @@
 GEM
   remote: https://rubygems.org/
   specs:
-    addressable (2.5.2)
-      public_suffix (>= 2.0.2, < 4.0)
-    asciidoctor (1.5.6.1)
+    addressable (2.7.0)
+      public_suffix (>= 2.0.2, < 5.0)
+    asciidoctor (2.0.10)
     colorator (1.1.0)
-    ffi (1.9.24)
+    concurrent-ruby (1.1.7)
+    em-websocket (0.5.1)
+      eventmachine (>= 0.12.9)
+      http_parser.rb (~> 0.6.0)
+    eventmachine (1.2.7)
+    ffi (1.13.1)
     forwardable-extended (2.6.0)
-    jekyll (3.6.3)
+    http_parser.rb (0.6.0)
+    i18n (1.8.5)
+      concurrent-ruby (~> 1.0)
+    jekyll (4.1.1)
       addressable (~> 2.4)
       colorator (~> 1.0)
-      jekyll-sass-converter (~> 1.0)
-      jekyll-watch (~> 1.1)
-      kramdown (~> 1.14)
+      em-websocket (~> 0.5)
+      i18n (~> 1.0)
+      jekyll-sass-converter (~> 2.0)
+      jekyll-watch (~> 2.0)
+      kramdown (~> 2.1)
+      kramdown-parser-gfm (~> 1.0)
       liquid (~> 4.0)
-      mercenary (~> 0.3.3)
+      mercenary (~> 0.4.0)
       pathutil (~> 0.9)
-      rouge (>= 1.7, < 3)
+      rouge (~> 3.0)
       safe_yaml (~> 1.0)
-    jekyll-sass-converter (1.5.0)
-      sass (~> 3.4)
-    jekyll-watch (1.5.0)
-      listen (~> 3.0, < 3.1)
-    json (2.1.0)
-    kramdown (1.16.1)
-    libv8 (3.16.14.19)
-    liquid (4.0.0)
-    listen (3.0.8)
-      rb-fsevent (~> 0.9, >= 0.9.4)
-      rb-inotify (~> 0.9, >= 0.9.7)
-    mercenary (0.3.6)
-    pathutil (0.16.0)
+      terminal-table (~> 1.8)
+    jekyll-sass-converter (2.1.0)
+      sassc (> 2.0.1, < 3.0)
+    jekyll-watch (2.2.1)
+      listen (~> 3.0)
+    json (2.3.1)
+    kramdown (2.3.0)
+      rexml
+    kramdown-parser-gfm (1.1.0)
+      kramdown (~> 2.0)
+    liquid (4.0.3)
+    listen (3.2.1)
+      rb-fsevent (~> 0.10, >= 0.10.3)
+      rb-inotify (~> 0.9, >= 0.9.10)
+    mercenary (0.4.0)
+    pathutil (0.16.2)
       forwardable-extended (~> 2.6)
-    public_suffix (2.0.5)
-    rb-fsevent (0.10.2)
-    rb-inotify (0.9.10)
-      ffi (>= 0.5.0, < 2)
-    rdoc (4.3.0)
-    redcarpet (3.4.0)
+    public_suffix (4.0.5)
+    rb-fsevent (0.10.4)
+    rb-inotify (0.10.1)
+      ffi (~> 1.0)
+    rdoc (6.2.1)
+    redcarpet (3.5.0)
     ref (2.0.0)
-    rouge (1.11.1)
-    safe_yaml (1.0.4)
-    sass (3.5.3)
-      sass-listen (~> 4.0.0)
-    sass-listen (4.0.0)
-      rb-fsevent (~> 0.9, >= 0.9.4)
-      rb-inotify (~> 0.9, >= 0.9.7)
-    therubyracer (0.12.3)
-      libv8 (~> 3.16.14.15)
-      ref
+    rexml (3.2.4)
+    rouge (3.22.0)
+    safe_yaml (1.0.5)
+    sassc (2.4.0)
+      ffi (~> 1.9)
+    terminal-table (1.8.0)
+      unicode-display_width (~> 1.1, >= 1.1.1)
     thread_safe (0.3.6)
-    tilt (2.0.8)
-    yajl-ruby (1.3.1)
+    tilt (2.0.10)
+    unicode-display_width (1.7.0)
+    yajl-ruby (1.4.1)
 
 PLATFORMS
   ruby
@@ -60,17 +72,12 @@ DEPENDENCIES
   asciidoctor
   jekyll
   json
-  libv8 (>= 3.16.14.14)
   rdoc
   redcarpet
   ref
-  therubyracer
   thread_safe
   tilt
   yajl-ruby (>= 1.3.1)
 
-RUBY VERSION
-   ruby 2.0.0p648
-
 BUNDLED WITH
-   1.16.0
+   2.1.4
diff --git a/docs/support/scripts/make_docs.sh b/docs/support/scripts/make_docs.sh
index 1b0b742..7738374 100755
--- a/docs/support/scripts/make_docs.sh
+++ b/docs/support/scripts/make_docs.sh
@@ -34,6 +34,21 @@ usage() {
   echo usage: "$0 --build_root <path> --output_subdir <relative path> [--site
<path to gh-pages checkout>]"
 }
 
+if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+  SED="sed -i -e"
+elif [[ "$OSTYPE" == "darwin"* ]]; then
+  echo "WARNING: The docs MUST NOT be built on Mac for publishing purposes.
+  Press 'y' to continue"
+  read -n 1
+  if [[ "$REPLY" != "y" ]]; then
+    exit 1;
+  fi
+  SED="sed -i~ -e"
+else
+  echo "Unsupported OS"
+  exit 1
+fi
+
 while [[ $# > 0 ]] ; do
     arg=$1
     case $arg in
@@ -113,10 +128,7 @@ BUNDLE="$GEM_PATH/bin/bundle"
 echo "Locally installing ruby gems needed to build docs."
 if [ ! -x "$BUNDLE" ]; then
   set -x
-  # The bundler install is pinned to an explicit version because versions after
-  # 2.0.0 require ruby >= 2.3.0 which is not available on many supported
-  # operating systems.
-  gem install --no-ri --no-rdoc -q --install-dir "$GEM_PATH" bundler -v '1.17.3'
+  gem install --no-document  -q --install-dir "$GEM_PATH" bundler
   set +x
 fi
 
@@ -163,7 +175,7 @@ for binary in ${binaries[@]}; do
     -o $GEN_DOC_DIR/${binary}_configuration_reference.adoc \
       $SOURCE_ROOT/docs/support/xsl/gflags_to_asciidoc.xsl \
     ${GEN_DOC_DIR}/$binary.xml
-  INCLUSIONS_SUPPORTED+="include::${binary}_configuration_reference.adoc[leveloffset=+1]\n"
+  INCLUSIONS_SUPPORTED+="include::${binary}_configuration_reference.adoc[leveloffset=+1]\\"$'\n'
 
   # Create the unsupported config reference
   xsltproc \
@@ -172,13 +184,13 @@ for binary in ${binaries[@]}; do
     -o $GEN_DOC_DIR/${binary}_configuration_reference_unsupported.adoc \
       $SOURCE_ROOT/docs/support/xsl/gflags_to_asciidoc.xsl \
     ${GEN_DOC_DIR}/$binary.xml
-  INCLUSIONS_UNSUPPORTED+="include::${binary}_configuration_reference_unsupported.adoc[leveloffset=+1]\n"
+  INCLUSIONS_UNSUPPORTED+="include::${binary}_configuration_reference_unsupported.adoc[leveloffset=+1]\\"$'\n'
 done
 
 # Add the includes to the configuration reference files, replacing the template lines
 cp $SOURCE_ROOT/docs/configuration_reference* $GEN_DOC_DIR/
-sed -i "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_SUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference.adoc
-sed -i "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_UNSUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference_unsupported.adoc
+$SED "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_SUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference.adoc
+$SED "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_UNSUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference_unsupported.adoc
 
 # Create tool references
 echo "Running kudu --helpxml"
@@ -202,7 +214,7 @@ ${GEN_DOC_DIR}/kudu.xml
 
 # Add the includes to the cli tools reference files, replacing the template lines
 cp $SOURCE_ROOT/docs/command_line_tools_reference.adoc $GEN_DOC_DIR/
-sed -i "s#@@TOOLS_REFERENCE@@#include::command_line_tools.adoc[leveloffset=+1]\n#" ${GEN_DOC_DIR}/command_line_tools_reference.adoc
+$SED "s#@@TOOLS_REFERENCE@@#include::command_line_tools.adoc[leveloffset=+1]\\"$'\n#' ${GEN_DOC_DIR}/command_line_tools_reference.adoc
 
 # If we're generating the web site, pass the template which causes us
 # to generate Jekyll templates instead of full HTML.


Mime
View raw message