drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sudhe...@apache.org
Subject drill git commit: Drill-4581: Extensive revisions to the Drill launch scripts.
Date Sat, 23 Jul 2016 00:42:49 GMT
Repository: drill
Updated Branches:
  refs/heads/master 4dac10347 -> 1f3c74dad


Drill-4581: Extensive revisions to the Drill launch scripts.

See DRILL-4581 and DRILL-4591 for an overview.
See DRILL-4581 for a detailed list of bugs fixed.
See DRILL-4591 for the motivation for the new "site" directory support.
Changes support DRILL-1170 (Drill-on-YARN).

Broad overview of changes:

* Extended the existing "config" directory concept to create a
"site" directory that holds all site-specific files, leaving the
Drill directory ($DRILL_HOME) to contain only Drill-provided files.
The site directory is handy for all Drill users because it eases
upgrades, but is necessary to simplify Drill-on-YARN deployments.
Use the --config (for backward compatibility) or --site (more
descriptive) option to point to the site directory.

* Moved distribution-specific settings, and Drill defaults, out of
drill-env.sh. Now, drill-env.sh contains only user settings, avoiding
the need to do multi-way merges on upgrades. Distribution-specific
files now reside in a new $DRILL_HOME/conf/distrib-env.sh file.

* Refactored the launch scripts to allow the bulk of setup to be shared
between the "classic" Drill daemon script (drillbit.sh) and the new
Drill-on-YARN scripts.

* Added a new "run" option to drillbit.sh to allow Drill to run as a
child process as needed by tools such as Apache Mesos.

* Changes ensure backward compatibility. Users of earlier releases can
upgrade to the release with this fix without doing anything special.
Drill will "just work." However, users can optionally clean up the
drill-env.sh script, optionally use the site directory, and so on.
However these upgrades are not required.

closes #547


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/1f3c74da
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/1f3c74da
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/1f3c74da

Branch: refs/heads/master
Commit: 1f3c74dad8681243f657025e2e911da13d2497ed
Parents: 4dac103
Author: Paul Rogers <progers@maprtech.com>
Authored: Tue Jul 12 16:19:28 2016 -0700
Committer: Sudheesh Katkam <skatkam@maprtech.com>
Committed: Fri Jul 22 17:03:21 2016 -0700

----------------------------------------------------------------------
 distribution/src/assemble/bin.xml               |  12 +-
 .../src/resources/core-site-example.xml         |  31 ++
 distribution/src/resources/core-site.xml        |  31 --
 distribution/src/resources/distrib-env.sh       |  20 ++
 distribution/src/resources/drill-config.sh      | 316 ++++++++++++----
 distribution/src/resources/drill-embedded       |   1 +
 distribution/src/resources/drill-env.sh         | 125 ++++++-
 distribution/src/resources/drillbit.sh          | 357 +++++++++----------
 distribution/src/resources/runbit               |  94 ++++-
 distribution/src/resources/sqlline              |  70 +++-
 10 files changed, 738 insertions(+), 319 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/assemble/bin.xml
----------------------------------------------------------------------
diff --git a/distribution/src/assemble/bin.xml b/distribution/src/assemble/bin.xml
index 12682e2..aad468e 100644
--- a/distribution/src/assemble/bin.xml
+++ b/distribution/src/assemble/bin.xml
@@ -286,11 +286,6 @@
       <outputDirectory>bin</outputDirectory>
     </file>
     <file>
-      <source>src/resources/dumpcat</source>
-      <fileMode>0755</fileMode>
-      <outputDirectory>bin</outputDirectory>
-    </file>
-    <file>
       <source>src/resources/drill-override.conf</source>
       <outputDirectory>conf</outputDirectory>
     </file>
@@ -304,11 +299,16 @@
       <outputDirectory>conf</outputDirectory>
     </file>
     <file>
+      <source>src/resources/distrib-env.sh</source>
+      <fileMode>0755</fileMode>
+      <outputDirectory>conf</outputDirectory>
+    </file>
+    <file>
       <source>src/resources/drill-override-example.conf</source>
       <outputDirectory>conf</outputDirectory>
     </file>
     <file>
-      <source>src/resources/core-site.xml</source>
+      <source>src/resources/core-site-example.xml</source>
       <outputDirectory>conf</outputDirectory>
     </file>
   </files>

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/core-site-example.xml
----------------------------------------------------------------------
diff --git a/distribution/src/resources/core-site-example.xml b/distribution/src/resources/core-site-example.xml
new file mode 100644
index 0000000..8a2ebe5
--- /dev/null
+++ b/distribution/src/resources/core-site-example.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements.  See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+    <property>
+        <name>fs.s3a.access.key</name>
+        <value>ENTER_YOUR_ACCESSKEY</value>
+    </property>
+
+    <property>
+        <name>fs.s3a.secret.key</name>
+        <value>ENTER_YOUR_SECRETKEY</value>
+    </property>
+
+</configuration>

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/core-site.xml
----------------------------------------------------------------------
diff --git a/distribution/src/resources/core-site.xml b/distribution/src/resources/core-site.xml
deleted file mode 100644
index 8a2ebe5..0000000
--- a/distribution/src/resources/core-site.xml
+++ /dev/null
@@ -1,31 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements.  See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License.  You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<configuration>
-
-    <property>
-        <name>fs.s3a.access.key</name>
-        <value>ENTER_YOUR_ACCESSKEY</value>
-    </property>
-
-    <property>
-        <name>fs.s3a.secret.key</name>
-        <value>ENTER_YOUR_SECRETKEY</value>
-    </property>
-
-</configuration>

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/distrib-env.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/distrib-env.sh b/distribution/src/resources/distrib-env.sh
new file mode 100644
index 0000000..1813b8d
--- /dev/null
+++ b/distribution/src/resources/distrib-env.sh
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is empty by default. Default Drill environment settings appear
+# in drill-config.sh. Distributions can replace this file with a
+# distribution-specific version that sets environment variables and options
+# specific to that distribution. Users should not put anything in this file;
+# put user options in drill-env.sh instead.

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/drill-config.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/drill-config.sh b/distribution/src/resources/drill-config.sh
index 3de6be4..a72cb84 100644
--- a/distribution/src/resources/drill-config.sh
+++ b/distribution/src/resources/drill-config.sh
@@ -20,6 +20,9 @@
 #
 # Environment Variables:
 #
+#   DRILL_HOME                 Drill home (defaults based on this
+#                              script's path.)
+#
 #   JAVA_HOME                  The java implementation to use.
 #
 #   DRILL_CLASSPATH            Extra Java CLASSPATH entries.
@@ -30,6 +33,23 @@
 #   HADOOP_HOME                Hadoop home
 #
 #   HBASE_HOME                 HBase home
+#
+# Variables may be set in one of four places:
+#
+#   Environment (per run)
+#   drill-env.sh (per site)
+#   distrib-env.sh (per distribution)
+#   drill-config.sh (this file, Drill defaults)
+#
+# Properties "inherit" from items lower on the list, and may be "overridden" by items
+# higher on the list. In the environment, just set the variable:
+#
+#   export FOO=value
+#
+# The three files must set values as shown below:
+#
+#   export FOO=${FOO:-"value"}
+#
 
 # resolve links - "${BASH_SOURCE-$0}" may be a softlink
 this="${BASH_SOURCE-$0}"
@@ -50,94 +70,252 @@ home=`cd "$bin/..">/dev/null; pwd`
 this="$home/bin/$script"
 
 # the root of the drill installation
-if [ -z "$DRILL_HOME" ]; then
-  DRILL_HOME="$home"
-fi
+DRILL_HOME=${DRILL_HOME:-$home}
 
-#check to see if the conf dir or drill home are given as an optional arguments
-while [ $# -gt 1 ]; do
-  if [ "--config" = "$1" ]; then
+# Standardize error messages
+
+fatal_error() {
+  echo "ERROR: $@" 1>&2
+  exit 1
+}
+
+# Check to see if the conf dir or drill home are given as an optional arguments
+# Arguments may appear anywhere on the command line. --site is an alias, better
+# specifies that the location contains all site-specific files, not just config.
+#
+# Remaining arguments go into the args array - use that instead of $@.
+
+args=()
+while [[ $# > 0 ]]
+do
+  arg="$1"
+  case "$arg" in
+  --site|--config)
     shift
-    confdir=$1
+    DRILL_CONF_DIR=$1
     shift
-    DRILL_CONF_DIR=$confdir
-  else
-    # Presume we are at end of options and break
-    break
-  fi
+    ;;
+  *)
+    args+=("$1")
+    shift
+    ;;
+  esac
 done
+export args
+
+# If config dir is given, it must exist.
+
+if [ -n "$DRILL_CONF_DIR" ]; then
+  if [[ ! -d "$DRILL_CONF_DIR" ]]; then
+    fatal_error "Config dir does not exist:" $DRILL_CONF_DIR
+  fi
+else
 
-# Allow alternate drill conf dir location.
-DRILL_CONF_DIR="${DRILL_CONF_DIR:-/etc/drill/conf}"
+  # Allow alternate drill conf dir location.
+  DRILL_CONF_DIR="/etc/drill/conf"
 
-if [ ! -d $DRILL_CONF_DIR ]; then
-  DRILL_CONF_DIR=$DRILL_HOME/conf
+  # Otherwise, use the default
+  if [[ ! -d "$DRILL_CONF_DIR" ]]; then
+    DRILL_CONF_DIR="$DRILL_HOME/conf"
+  fi
+fi
+
+# However we got the config dir, it must contain a config
+# file, and that file must be readable.
+# Most files are optional, so check the one that is required:
+# drill-override.conf.
+
+testFile="$DRILL_CONF_DIR/drill-override.conf"
+if [[ ! -a "$testFile" ]]; then
+  fatal_error "Drill config file missing: $testFile -- Wrong config dir?"
+fi
+if [[ ! -r "$testFile" ]]; then
+  fatal_error "Drill config file not readable: $testFile - Wrong user?"
+fi
+
+# Set Drill-provided defaults here. Do not put Drill defaults
+# in the distribution or user environment config files.
+
+# The SQLline client does not need the code cache.
+
+export SQLLINE_JAVA_OPTS=${SQLLINE_JAVA_OPTS:-"-XX:MaxPermSize=512M"}
+
+# Class unloading is disabled by default in Java 7
+# http://hg.openjdk.java.net/jdk7u/jdk7u60/hotspot/file/tip/src/share/vm/runtime/globals.hpp#l1622
+export SERVER_GC_OPTS="$SERVER_GC_OPTS -XX:+CMSClassUnloadingEnabled -XX:+UseG1GC"
+
+# No GC options by default for SQLLine
+export CLIENT_GC_OPTS=${CLIENT_GC_OPTS:-""}
+
+# Source the optional drill-env.sh for any user configured values.
+# We read the file only in the $DRILL_CONF_DIR, which might be a
+# site-specific folder. By design, we do not search both the site
+# folder and the $DRILL_HOME/conf folder; we look in just the one
+# identified by $DRILL_CONF_DIR.
+#
+# Note: the env files must set properties as follows for "inheritance"
+# to work correctly:
+#
+# export FOO=${FOO:-"value"}
+
+drillEnv="$DRILL_CONF_DIR/drill-env.sh"
+if [ -r "$drillEnv" ]; then
+  . "$drillEnv"
 fi
 
-# Source drill-env.sh for any user configured values
-. "${DRILL_CONF_DIR}/drill-env.sh"
+# Source distrib-env.sh for any distribution-specific settings.
+# distrib-env.sh is optional; it is created by some distribution installers
+# that need distribution-specific settings.
+
+distribEnv="$DRILL_HOME/conf/distrib-env.sh"
+if [ -r "$distribEnv" ]; then
+  . "$distribEnv"
+fi
+
+# Default memory settings if none provided by the environment or
+# above config files.
+# The Drillbit needs a large code cache.
+
+export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"8G"}
+export DRILL_HEAP=${DRILL_HEAP:-"4G"}
+export DRILLBIT_MAX_PERM=${DRILLBIT_MAX_PERM:-"512M"}
+export DRILLBIT_CODE_CACHE_SIZE=${DRILLBIT_CODE_CACHE_SIZE:-"1G"}
+
+export DRILLBIT_OPTS="-Xms$DRILL_HEAP -Xmx$DRILL_HEAP -XX:MaxDirectMemorySize=$DRILL_MAX_DIRECT_MEMORY"
+export DRILLBIT_OPTS="$DRILLBIT_OPTS -XX:ReservedCodeCacheSize=$DRILLBIT_CODE_CACHE_SIZE -Ddrill.exec.enable-epoll=true"
+export DRILLBIT_OPTS="$DRILLBIT_OPTS -XX:MaxPermSize=$DRILLBIT_MAX_PERM"
+
+# Under YARN, the log directory is usually YARN-provided. Replace any
+# value that may have been set in drill-env.sh.
+
+if [ -n "$DRILL_YARN_LOG_DIR" ]; then
+  DRILL_LOG_DIR="$DRILL_YARN_LOG_DIR"
+fi
 
 # get log directory
-if [ "x${DRILL_LOG_DIR}" = "x" ]; then
-  export DRILL_LOG_DIR=/var/log/drill
+if [ -z "$DRILL_LOG_DIR" ]; then
+  # Try the optional location
+  DRILL_LOG_DIR=/var/log/drill
+  if [[ ! -d "$DRILL_LOG_DIR" && ! -w "$DRILL_LOG_DIR" ]]; then
+    # Default to the drill home folder. Create the directory
+    # if not present.
+
+    DRILL_LOG_DIR=$DRILL_HOME/log
+  fi
 fi
 
-touch "$DRILL_LOG_DIR/sqlline.log" &> /dev/null
-TOUCH_EXIT_CODE=$?
-if [ "$TOUCH_EXIT_CODE" = "0" ]; then
-  if [ "x$DRILL_LOG_DEBUG" = "x1" ]; then
-    echo "Drill log directory: $DRILL_LOG_DIR"
+# Regardless of how we got the directory, it must exist
+# and be writable.
+
+mkdir -p "$DRILL_LOG_DIR"
+if [[ ! -d "$DRILL_LOG_DIR" && ! -w "$DRILL_LOG_DIR" ]]; then
+  fatal_error "Log directory does not exist or is not writable: $DRILL_LOG_DIR"
+fi
+
+# Store the pid file in Drill home by default, else in the location
+# provided in drill-env.sh.
+
+export DRILL_PID_DIR=${DRILL_PID_DIR:-$DRILL_HOME}
+
+# Prepare log file prefix and the main Drillbit log file.
+
+export DRILL_LOG_PREFIX="$DRILL_LOG_DIR/drillbit"
+export DRILLBIT_LOG_PATH="${DRILL_LOG_PREFIX}.log"
+
+# Class path construction.
+
+# Add Drill conf folder at the beginning of the classpath
+CP="$DRILL_CONF_DIR"
+
+# If both user and YARN-provided Java lib paths exist,
+# combine them.
+
+if [ -n "$DOY_JAVA_LIB_PATH" ]; then
+  if [ -z "$DRILL_JAVA_LIB_PATH" ]; then
+    export DRILL_JAVA_LIB_PATH="$DOY_JAVA_LIB_PATH"
+  else
+    export DRILL_JAVA_LIB_PATH="$DOY_JAVA_LIB_PATH:$DRILL_JAVA_LIB_PATH"
   fi
-  DRILL_LOG_DIR_FALLBACK=0
-else
-  #Force DRILL_LOG_DIR to fall back
-  DRILL_LOG_DIR_FALLBACK=1
 fi
 
-if [ ! -d "$DRILL_LOG_DIR" ] || [ "$DRILL_LOG_DIR_FALLBACK" = "1" ]; then
-  if [ "x$DRILL_LOG_DEBUG" = "x1" ]; then
-    echo "Drill log directory $DRILL_LOG_DIR does not exist or is not writable, defaulting to $DRILL_HOME/log"
+# Add the lib directory to the library path, if it exists.
+
+libDir="$DRILL_CONF_DIR/lib"
+if [ -d "$libDir" ]; then
+  if [ -z "$DRILL_JAVA_LIB_PATH" ]; then
+    export DRILL_JAVA_LIB_PATH="$libDir"
+  else
+    export DRILL_JAVA_LIB_PATH="$libDir:$DRILL_JAVA_LIB_PATH"
   fi
-  DRILL_LOG_DIR=$DRILL_HOME/log
-  mkdir -p $DRILL_LOG_DIR
 fi
 
-# Add Drill conf folder at the beginning of the classpath
-CP=$DRILL_CONF_DIR
+# Add $DRILL_HOME/conf if the user has provided their own
+# site configuration directory.
+# Ensures we pick up the default logback.xml, etc. if the
+# user does not provide their own.
+# Also, set a variable to remember that the config dir
+# is non-default, which is needed later.
+
+if [[ ! "$DRILL_CONF_DIR" -ef "$DRILL_HOME/conf" ]]; then
+  export DRILL_SITE_DIR="$DRILL_CONF_DIR"
+  CP="$CP:$DRILL_HOME/conf"
+fi
 
 # Followed by any user specified override jars
-if [ "${DRILL_CLASSPATH_PREFIX}x" != "x" ]; then
-  CP=$CP:$DRILL_CLASSPATH_PREFIX
+if [ -n "$DRILL_CLASSPATH_PREFIX" ]; then
+  CP="$CP:$DRILL_CLASSPATH_PREFIX"
 fi
 
 # Next Drill core jars
-CP=$CP:$DRILL_HOME/jars/*
+if [ -n "$DRILL_TOOL_CP" ]; then
+  CP="$CP:$DRILL_TOOL_CP"
+fi
+CP="$CP:$DRILL_HOME/jars/*"
 
 # Followed by Drill override dependency jars
-CP=$CP:$DRILL_HOME/jars/ext/*
+CP="$CP:$DRILL_HOME/jars/ext/*"
 
 # Followed by Hadoop's jar
-if [ "${HADOOP_CLASSPATH}x" != "x" ]; then
-  CP=$CP:$HADOOP_CLASSPATH
+if [ -n "$HADOOP_CLASSPATH" ]; then
+  CP="$CP:$HADOOP_CLASSPATH"
 fi
 
-# Followed by HBase' jar
-if [ "${HBASE_CLASSPATH}x" != "x" ]; then
-  CP=$CP:$HBASE_CLASSPATH
+# Followed by HBase's jar
+if [ -n "$HBASE_CLASSPATH" ]; then
+  CP="$CP:$HBASE_CLASSPATH"
 fi
 
-# Followed by Drill other dependency jars
-CP=$CP:$DRILL_HOME/jars/3rdparty/*
-CP=$CP:$DRILL_HOME/jars/classb/*
+# Generalized extension path (use this for new deployments instead
+# of the specialized HADOOP_ and HBASE_CLASSPATH variables.)
+# Drill-on-YARN uses this variable to avoid the need to add more
+# XYX_CLASSPATH variables as we integrate with other external
+# systems.
 
-# Finally any user specified 
-if [ "${DRILL_CLASSPATH}x" != "x" ]; then
-  CP=$CP:$DRILL_CLASSPATH
+if [ -n "$EXTN_CLASSPATH" ]; then
+  CP="$CP:$EXTN_CLASSPATH"
 fi
 
-# Newer versions of glibc use an arena memory allocator that causes virtual
-# memory usage to explode. Tune the variable down to prevent vmem explosion.
-export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4}
+# Followed by Drill's other dependency jars
+
+CP="$CP:$DRILL_HOME/jars/3rdparty/*"
+CP="$CP:$DRILL_HOME/jars/classb/*"
+
+# Finally any user specified
+# Allow user jars to appear in $DRILL_CONF_DIR/jars to avoid mixing
+# user and Drill distribution jars.
+
+if [ -d "$DRILL_CONF_DIR/jars" ]; then
+  CP="$CP:$DRILL_CONF_DIR/jars/*"
+fi
+
+# The Drill classpath is a catch-all for any other jars that
+# a specific run might need. The use of this variable is for jars that
+# are not in a Drill directory; that means the jars must exist on every
+# node in the cluster.
+
+if [ -n "$DRILL_CLASSPATH" ]; then
+  CP="$CP:$DRILL_CLASSPATH"
+fi
 
 # Test for cygwin
 is_cygwin=false
@@ -152,22 +330,18 @@ if [ -z "$JAVA_HOME" ]; then
     while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
       DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
       SOURCE="$(readlink "$SOURCE")"
-      [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+      # if $SOURCE was a relative symlink, we need to resolve it relative
+      # to the path where the symlink file was located
+      [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
     done
     JAVA_HOME="$( cd -P "$( dirname "$SOURCE" )" && cd .. && pwd )"
   fi
   # if we didn't set it
   if [ -z "$JAVA_HOME" ]; then
-    cat 1>&2 <<EOF
-+======================================================================+
-|      Error: JAVA_HOME is not set and Java could not be found         |
-+----------------------------------------------------------------------+
-| Apache Drill requires Java 1.7 or later.                             |
-+======================================================================+
-EOF
-    exit 1
+    fatal_error "JAVA_HOME is not set and Java could not be found"
   fi
 fi
+
 # Now, verify that 'java' binary exists and is suitable for Drill.
 if $is_cygwin; then
   JAVA_BIN="java.exe"
@@ -176,14 +350,13 @@ else
 fi
 JAVA=`find -L "$JAVA_HOME" -name $JAVA_BIN -type f | head -n 1`
 if [ ! -e "$JAVA" ]; then
-  echo "Java not found at JAVA_HOME=$JAVA_HOME."
-  exit 1
+  fatal_error "Java not found at JAVA_HOME=$JAVA_HOME."
 fi
+
 # Ensure that Java version is at least 1.7
 "$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
 if [ $? -eq 0 ]; then
-  echo "Java 1.7 or later is required to run Apache Drill."
-  exit 1
+  fatal_error "Java 1.7 or later is required to run Apache Drill."
 fi
 
 # Adjust paths for CYGWIN
@@ -193,21 +366,24 @@ if $is_cygwin; then
   DRILL_LOG_DIR=`cygpath -w "$DRILL_LOG_DIR"`
   CP=`cygpath -w -p "$CP"`
   if [ -z "$HADOOP_HOME" ]; then
-    HADOOP_HOME=${DRILL_HOME}/winutils
+    export HADOOP_HOME=${DRILL_HOME}/winutils
   fi
 fi
 
 # make sure allocator chunks are done as mmap'd memory (and reduce arena overhead)
-export MALLOC_ARENA_MAX=4
+# Newer versions of glibc use an arena memory allocator that causes virtual
+# memory usage to explode. Tune the variable down to prevent vmem explosion.
+export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4}
 export MALLOC_MMAP_THRESHOLD_=131072
 export MALLOC_TRIM_THRESHOLD_=131072
 export MALLOC_TOP_PAD_=131072
 export MALLOC_MMAP_MAX_=65536
 
 # Variables exported form this script
-export HADOOP_HOME
 export is_cygwin
 export DRILL_HOME
 export DRILL_CONF_DIR
 export DRILL_LOG_DIR
 export CP
+export JAVA_HOME
+export JAVA

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/drill-embedded
----------------------------------------------------------------------
diff --git a/distribution/src/resources/drill-embedded b/distribution/src/resources/drill-embedded
index d66ba7b..51485f8 100755
--- a/distribution/src/resources/drill-embedded
+++ b/distribution/src/resources/drill-embedded
@@ -19,4 +19,5 @@ bin=`dirname "${BASH_SOURCE-$0}"`
 bin=`cd "$bin">/dev/null; pwd`
 
 # Start a sqlline session with an embedded Drillbit
+export DRILL_EMBEDDED=1
 exec ${bin}/sqlline -u "jdbc:drill:zk=local" "$@"

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/drill-env.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/drill-env.sh b/distribution/src/resources/drill-env.sh
index 53a3cb7..9e11458 100644
--- a/distribution/src/resources/drill-env.sh
+++ b/distribution/src/resources/drill-env.sh
@@ -12,12 +12,125 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+#-----------------------------------------------------------------------------
 
-DRILL_MAX_DIRECT_MEMORY="8G"
-DRILL_HEAP="4G"
+# This file provides a variety of site-specific settings to control Drill
+# launch settings. These are settings required when launching the Drillbit
+# or sqlline processes using Java. Some settings are for both, some for one
+# or the other.
+#
+# Variables may be set in one of four places:
+#
+#   Environment (per run)
+#   drill-env.sh (this file, per site)
+#   distrib-env.sh (per distribution)
+#   drill-config.sh (Drill defaults)
+#
+# Properties "inherit" from items lower on the list, and may be "overridden" by items
+# higher on the list. In the environment, just set the variable:
+#
+#   export FOO=value
+#
+# To support inheritance from the environment, you must set values as shown below:
+#
+#   export FOO=${FOO:-"value"}
+#
+# or a more specialized form.
+
+# Amount of heap memory for the Drillbit process. Values are those supported by
+# the Java -Xms option. The default is 4G.
+
+#export DRILL_HEAP=${DRILL_HEAP:-"4G"}
+
+# Maximum amount of direct memory to allocate to the Drillbit in the format
+# supported by -XX:MaxDirectMemorySize. Default is 8G.
+
+#export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"8G"}
+
+# Value for the JVM -XX:MaxPermSize option for the Drillbit. Default is 512M.
+
+#export DRILLBIT_MAX_PERM=${DRILLBIT_MAX_PERM:-"512M"}
+
+# Native library path passed to Java. Note: use this form instead
+# of the old form of DRILLBIT_JAVA_OPTS="-Djava.library.path=<dir>"
+# The old form is not compatible with Drill-on-YARN.
+
+# export DRILL_JAVA_LIB_PATH="<lib1>:<lib2>"
+
+# Value for the code cache size for the Drillbit. Because the Drillbit generates
+# code, it benefits from a large cache. Default is 1G.
+
+#export DRILLBIT_CODE_CACHE_SIZE=${DRILLBIT_CODE_CACHE_SIZE:-"1G"}
+
+# Location to place Drill logs. Set to $DRILL_HOME/log by default.
+
+#export DRILL_LOG_DIR=${DRILL_LOG_DIR:-$DRILL_HOME/conf}
+
+# Location to place the Drillbit pid file when running as a daemon using
+# drillbit.sh start.
+# Set to $DRILL_HOME by default.
+
+#export DRILL_PID_DIR=${DRILL_PID_DIR:-$DRILL_HOME}
+
+# Custom JVM arguments to pass to the both the Drillbit and sqlline. Typically
+# used to override system properties as shown below. Empty by default.
+
+#export DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS -Dproperty=value"
+
+# As above, but only for the Drillbit. Empty by default.
+
+#export DRILLBIT_JAVA_OPTS="$DRILLBIT_JAVA_OPTS -Dproperty=value"
+
+# Process priority (niceness) for the Drillbit when running as a daemon.
+# Defaults to 0.
+
+#export DRILL_NICENESS=${DRILL_NICENESS:-0}
+
+# Custom class path for Drill. In general, you should put your custom libraries into
+# your site directory's jars subfolder ($DRILL_HOME/conf/jars by default, but can be
+# customized with DRILL_CONF_DIR or the --config argument. But, if you must reference
+# jar files in other locations, you can add them here. These jars are added to the
+# Drill classpath after all Drill-provided jars. Empty by default.
+
+# custom="/your/path/here:/your/second/path"
+# if [ -z "$DRILL_CLASSPATH" ]; then
+#   export DRILL_CLASSPATH=${DRILL_CLASSPATH:$custom}
+# else
+#   export DRILL_CLASSPATH="$custom"
+# fi
+
+# Extension classpath for things like HADOOP, HBase and so on. Set as above.
+
+# EXTN_CLASSPATH=...
+
+# Note that one environment variable can't be set here: DRILL_CONF_DIR.
+# That variable tells Drill the location of this file, so this file can't
+# set it. Instead, you can set it in the environment, or using the
+# --config option of drillbit.sh or sqlline.
+
+#-----------------------------------------------------------------------------
+# The following are "advanced" options seldom used except when diagnosing
+# complex issues.
+#
+# The prefix class path appears before any Drill-provided classpath entries.
+# Use it to override Drill jars with specialized versions.
+
+#export DRILL_CLASSPATH_PREFIX=...
+
+# Enable garbage collection logging in the Drillbit. Logging goes to
+# $DRILL_LOG_DIR/drillbit.gc. A value of 1 enables logging, all other values
+# (including the default unset value) disables logging.
+
+#export SERVER_LOG_GC=${SERVER_LOG_GC:-1}
+
+# JVM options when running the sqlline Drill client. For example, adjust the
+# JVM heap memory here. These are used ONLY in non-embedded mode; these
+# are client-only settings. (The Drillbit settings are used when Drill
+# is embedded.)
+
+#export SQLLINE_JAVA_OPTS="-XX:MaxPermSize=512M"
 
-export DRILL_JAVA_OPTS="-Xms$DRILL_HEAP -Xmx$DRILL_HEAP -XX:MaxDirectMemorySize=$DRILL_MAX_DIRECT_MEMORY -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=1G -Ddrill.exec.enable-epoll=true"
+# Arguments passed to sqlline (the Drill shell) at all times: whether
+# Drill is embedded in Sqlline or not.
 
-# Class unloading is disabled by default in Java 7
-# http://hg.openjdk.java.net/jdk7u/jdk7u60/hotspot/file/tip/src/share/vm/runtime/globals.hpp#l1622
-export SERVER_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseG1GC "
+#export DRILL_SHELL_JAVA_OPTS="..."
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/drillbit.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/drillbit.sh b/distribution/src/resources/drillbit.sh
index 2b324ad..5f49ac7 100755
--- a/distribution/src/resources/drillbit.sh
+++ b/distribution/src/resources/drillbit.sh
@@ -1,247 +1,240 @@
 #!/usr/bin/env bash
+# Copyright 2013 The Apache Software Foundation
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 #
-#/**
-# * Copyright 2013 The Apache Software Foundation
-# *
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements.  See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership.  The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License.  You may obtain a copy of the License at
-# *
-# *     http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-# 
 # Environment Variables
 #
-#   DRILL_CONF_DIR   Alternate drill conf dir. Default is ${DRILL_HOME}/conf.
-#   DRILL_LOG_DIR    Where log files are stored.  PWD by default.
-#   DRILL_PID_DIR    The pid files are stored. /tmp by default.
-#   DRILL_IDENT_STRING   A string representing this instance of drillbit. $USER by default
-#   DRILL_NICENESS The scheduling priority for daemons. Defaults to 0.
-#   DRILL_STOP_TIMEOUT  Time, in seconds, after which we kill -9 the server if it has not stopped.
-#                        Default 120 seconds.
+#   DRILL_CONF_DIR      Alternate drill conf dir. Default is ${DRILL_HOME}/conf.
+#   DRILL_LOG_DIR       Where log files are stored. Default is /var/log/drill if
+#                       that exists, else $DRILL_HOME/log
+#   DRILL_PID_DIR       The pid files are stored. $DRILL_HOME by default.
+#   DRILL_IDENT_STRING  A string representing this instance of drillbit.
+#                       $USER by default
+#   DRILL_NICENESS      The scheduling priority for daemons. Defaults to 0.
+#   DRILL_STOP_TIMEOUT  Time, in seconds, after which we kill -9 the server if
+#                       it has not stopped.
+#                       Default 120 seconds.
+#   SERVER_LOG_GC       Set to "1" to enable Java garbage collector logging.
+#
+# See also the environment variables defined in drill-config.sh
+# and runbit. Most of the above can be set in drill-env.sh for
+# each site.
+#
+# Modeled after $HADOOP_HOME/bin/hadoop-daemon.sh
+#
+# Usage:
+#
+# drillbit.sh [--config conf-dir] cmd [arg1 arg2 ...]
+#
+# The configuration directory, if provided, must exist and contain a Drill
+# configuration file. The option takes precedence over the
+# DRILL_CONF_DIR environment variable.
 #
-# Modelled after $HADOOP_HOME/bin/hadoop-daemon.sh
+# The command is one of: start|stop|status|restart|run
+#
+# Additional arguments are passed as JVM options to the Drill-bit.
+# They typically are of the form:
+#
+# -Dconfig-var=value
+#
+# Where config-var is a fully expanded form of a configuration variable.
+# The value overrides any value in the user or Drill configuration files.
+
+usage="Usage: drillbit.sh [--config|--site <site-dir>]\
+ (start|stop|status|restart|run) [args]"
+
+bin=`dirname "${BASH_SOURCE-$0}"`
+bin=`cd "$bin">/dev/null; pwd`
+
+base=`basename "${BASH_SOURCE-$0}"`
+command=${base/.*/}
 
-usage="Usage: drillbit.sh [--config <conf-dir>]\
- (start|stop|status|restart|autorestart)"
+# Setup environment. This parses, and removes, the
+# options --config conf-dir parameters.
+
+. "$bin/drill-config.sh"
 
 # if no args specified, show usage
-if [ $# -lt 1 ]; then
+if [ ${#args[@]} = 0 ]; then
   echo $usage
   exit 1
 fi
 
-bin=`dirname "${BASH_SOURCE-$0}"`
-bin=`cd "$bin">/dev/null; pwd`
-
-. "$bin"/drill-config.sh
+# Get command. all other args are JVM args, typically properties.
+startStopStatus="${args[0]}"
+args[0]=''
+export args
 
-# get arguments
-startStopStatus=$1
-shift
-
-command=drillbit
-shift
+# Set default scheduling priority
+DRILL_NICENESS=${DRILL_NICENESS:-0}
 
-waitForProcessEnd() {
+waitForProcessEnd()
+{
   pidKilled=$1
   commandName=$2
   processedAt=`date +%s`
+  origcnt=${DRILL_STOP_TIMEOUT:-120}
   while kill -0 $pidKilled > /dev/null 2>&1;
    do
      echo -n "."
      sleep 1;
      # if process persists more than $DRILL_STOP_TIMEOUT (default 120 sec) no mercy
-     if [ $(( `date +%s` - $processedAt )) -gt ${DRILL_STOP_TIMEOUT:-120} ]; then
+     if [ $(( `date +%s` - $processedAt )) -gt $origcnt ]; then
        break;
      fi
-   done
+  done
+  echo
   # process still there : kill -9
   if kill -0 $pidKilled > /dev/null 2>&1; then
-    echo -n force stopping $commandName with kill -9 $pidKilled
+    echo "$commandName did not complete after $origcnt seconds, killing with kill -9 $pidKilled"
     $JAVA_HOME/bin/jstack -l $pidKilled > "$logout" 2>&1
     kill -9 $pidKilled > /dev/null 2>&1
   fi
-  # Add a CR after we're done w/ dots.
-  echo
 }
 
-drill_rotate_log ()
+check_before_start()
 {
-    log=$1;
-    num=5;
-    if [ -n "$2" ]; then
-    num=$2
-    fi
-    if [ -f "$log" ]; then # rotate logs
-    while [ $num -gt 1 ]; do
-        prev=`expr $num - 1`
-        [ -f "$log.$prev" ] && mv -f "$log.$prev" "$log.$num"
-        num=$prev
-    done
-    mv -f "$log" "$log.$num";
+  #check that the process is not running
+  mkdir -p "$DRILL_PID_DIR"
+  if [ -f $pid ]; then
+    if kill -0 `cat $pid` > /dev/null 2>&1; then
+      echo "$command is already running as process `cat $pid`.  Stop it first."
+      exit 1
     fi
+  fi
 }
 
-check_before_start(){
-    #ckeck if the process is not running
-    mkdir -p "$DRILL_PID_DIR"
-    if [ -f $pid ]; then
-      if kill -0 `cat $pid` > /dev/null 2>&1; then
-        echo $command running as process `cat $pid`.  Stop it first.
-        exit 1
-      fi
+wait_until_done ()
+{
+  p=$1
+  cnt=${DRILLBIT_TIMEOUT:-300}
+  origcnt=$cnt
+  while kill -0 $p > /dev/null 2>&1; do
+    if [ $cnt -gt 1 ]; then
+      cnt=`expr $cnt - 1`
+      sleep 1
+    else
+      echo "Process did not complete after $origcnt seconds, killing."
+      kill -9 $p
+      exit 1
     fi
+  done
+  return 0
 }
 
-wait_until_done ()
+start_bit ( )
 {
-    p=$1
-    cnt=${DRILLBIT_TIMEOUT:-300}
-    origcnt=$cnt
-    while kill -0 $p > /dev/null 2>&1; do
-      if [ $cnt -gt 1 ]; then
-        cnt=`expr $cnt - 1`
-        sleep 1
-      else
-        echo "Process did not complete after $origcnt seconds, killing."
-        kill -9 $p
-        exit 1
-      fi
-    done
-    return 0
+  check_before_start
+  echo "Starting $command, logging to $logout"
+  echo "`date` Starting $command on `hostname`" >> "$DRILLBIT_LOG_PATH"
+  echo "`ulimit -a`" >> "$DRILLBIT_LOG_PATH" 2>&1
+  nohup nice -n $DRILL_NICENESS "$DRILL_HOME/bin/runbit" exec ${args[@]} >> "$logout" 2>&1 &
+  echo $! > $pid
+  sleep 1
 }
 
-# get log directory
-if [ "$DRILL_LOG_DIR" = "" ]; then
-  export DRILL_LOG_DIR=/var/log/drill
-fi
-mkdir -p "$DRILL_LOG_DIR"
-
-if [ "$DRILL_PID_DIR" = "" ]; then
-  DRILL_PID_DIR=$DRILL_HOME
-fi
-
-# Some variables
-# Work out java location so can print version into log.
-if [ "$JAVA_HOME" != "" ]; then
-  #echo "run java in $JAVA_HOME"
-  JAVA_HOME=$JAVA_HOME
-fi
-if [ "$JAVA_HOME" = "" ]; then
-  echo "Error: JAVA_HOME is not set."
-  exit 1
-fi
+stop_bit ( )
+{
+  if [ -f $pid ]; then
+    pidToKill=`cat $pid`
+    # kill -0 == see if the PID exists
+    if kill -0 $pidToKill > /dev/null 2>&1; then
+      echo "Stopping $command"
+      echo "`date` Terminating $command pid $pidToKill" >> "$DRILLBIT_LOG_PATH"
+      kill $pidToKill > /dev/null 2>&1
+      waitForProcessEnd $pidToKill $command
+      retval=0
+    else
+      retval=$?
+      echo "No $command to stop because kill -0 of pid $pidToKill failed with status $retval"
+    fi
+    rm $pid > /dev/null 2>&1
+  else
+    echo "No $command to stop because no pid file $pid"
+    retval=1
+  fi
+  return $retval
+}
 
-JAVA=$JAVA_HOME/bin/java
-export DRILL_LOG_PREFIX=drillbit
-export DRILL_LOGFILE=$DRILL_LOG_PREFIX.log
-export DRILL_OUTFILE=$DRILL_LOG_PREFIX.out
-export DRILL_QUERYFILE=${DRILL_LOG_PREFIX}_queries.json
-loggc=$DRILL_LOG_DIR/$DRILL_LOG_PREFIX.gc
-loglog="${DRILL_LOG_DIR}/${DRILL_LOGFILE}"
-logout="${DRILL_LOG_DIR}/${DRILL_OUTFILE}"
-logqueries="${DRILL_LOG_DIR}/${DRILL_QUERYFILE}"
 pid=$DRILL_PID_DIR/drillbit.pid
-
-export DRILLBIT_LOG_PATH=$loglog
-export DRILLBIT_QUERY_LOG_PATH=$logqueries
-
-if [ -n "$SERVER_GC_OPTS" ]; then
-  export SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}"}
-fi
-if [ -n "$CLIENT_GC_OPTS" ]; then
-  export CLIENT_GC_OPTS=${CLIENT_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}"}
-fi
-
-# Set default scheduling priority
-if [ "$DRILL_NICENESS" = "" ]; then
-    export DRILL_NICENESS=0
-fi
+logout="${DRILL_LOG_PREFIX}.out"
 
 thiscmd=$0
-args=$@
 
 case $startStopStatus in
 
 (start)
-    check_before_start
-    echo starting $command, logging to $logout
-    nohup $thiscmd internal_start $command $args < /dev/null >> ${logout} 2>&1  &
-    sleep 1;
+  start_bit
   ;;
 
-(internal_start)
-    drill_rotate_log $loggc
-    # Add to the command log file vital stats on our environment.
-    echo "`date` Starting $command on `hostname`" >> $loglog
-    echo "`ulimit -a`" >> $loglog 2>&1
-    nice -n $DRILL_NICENESS "$DRILL_HOME"/bin/runbit \
-        $command "$@" start >> "$logout" 2>&1 &
-    echo $! > $pid
-    wait
+(run)
+  # Launch Drill as a child process. Does not redirect stderr or stdout.
+  # Does not capture the Drillbit pid.
+  # Use this when launching Drill from your own script that manages the
+  # process, such as (roll-your-own) YARN, Mesos, supervisord, etc.
+
+  echo "`date` Starting $command on `hostname`"
+  echo "`ulimit -a`"
+  $DRILL_HOME/bin/runbit exec ${args[@]}
   ;;
 
 (stop)
-    rm -f "$DRILL_START_FILE"
-    if [ -f $pid ]; then
-      pidToKill=`cat $pid`
-      # kill -0 == see if the PID exists
-      if kill -0 $pidToKill > /dev/null 2>&1; then
-        echo stopping $command
-        echo "`date` Terminating $command" pid $pidToKill>> $loglog
-        kill $pidToKill > /dev/null 2>&1
-        waitForProcessEnd $pidToKill $command
-        rm $pid
-      else
-        retval=$?
-        echo no $command to stop because kill -0 of pid $pidToKill failed with status $retval
-      fi
-    else
-      echo no $command to stop because no pid file $pid
-    fi
+  stop_bit
+  exit $?
   ;;
 
 (restart)
-    # stop the command
-    $thiscmd --config "${DRILL_CONF_DIR}" stop $command $args &
-    wait_until_done $!
-    # wait a user-specified sleep period
-    sp=${DRILL_RESTART_SLEEP:-3}
-    if [ $sp -gt 0 ]; then
-      sleep $sp
-    fi
-    # start the command
-    $thiscmd --config "${DRILL_CONF_DIR}" start $command $args &
-    wait_until_done $!
+  # stop the command
+  stop_bit
+  # wait a user-specified sleep period
+  sp=${DRILL_RESTART_SLEEP:-3}
+  if [ $sp -gt 0 ]; then
+    sleep $sp
+  fi
+  # start the command
+  start_bit
   ;;
 
 (status)
-
-    if [ -f $pid ]; then
-      TARGET_PID=`cat $pid`
-      if kill -0 $TARGET_PID > /dev/null 2>&1; then
-        echo $command is running.
-        exit 0
-      else
-        echo $pid file is present but $command not running.
-        exit 1
-      fi
+  if [ -f $pid ]; then
+    TARGET_PID=`cat $pid`
+    if kill -0 $TARGET_PID > /dev/null 2>&1; then
+      echo "$command is running."
     else
-      echo $command not running.
-      exit 2
+      echo "$pid file is present but $command is not running."
+      exit 1
     fi
-    ;;
+  else
+    echo "$command is not running."
+    exit 1
+  fi
+  ;;
+
+(debug)
+  # Undocumented command to print out environment and Drillbit
+  # command line after all adjustments.
+
+  echo "command: $command"
+  echo "args: ${args[@]}"
+  echo "cwd:" `pwd`
+  # Print Drill command line
+  "$DRILL_HOME/bin/runbit" debug ${args[@]}
+  ;;
 
 (*)
   echo $usage

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/runbit
----------------------------------------------------------------------
diff --git a/distribution/src/resources/runbit b/distribution/src/resources/runbit
index 8c54b1a..1e36f86 100755
--- a/distribution/src/resources/runbit
+++ b/distribution/src/resources/runbit
@@ -15,11 +15,95 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-bin=`dirname "${BASH_SOURCE-$0}"`
-bin=`cd "$bin">/dev/null; pwd`
+# Invoke Drill using Java. Command line arguments are assumed to be
+# of the form
+#   exec|debug
+# The debug option simply prints the command line option, but does not
+# run Drill. Anything else runs Drill. (Command line options passed to
+# drillbit.sh are passed into this script in the args[] variable, see
+# below.)
+#
+# Environment Variables
+#
+#   SERVER_LOG_GC           Set to "1" to enable Java garbage collector
+#                           logging.
+#   DRILL_LOG_PREFIX        Path and name prefix for log files.
+#                           (Set in drill-config.sh.)
+#   DRILLBIT_LOG_PATH       Path to the Drillbit log file.
+#                           (Set in drill-config.sh.)
+#   DRILL_JAVA_OPTS         Optional JVM arguments such as system
+#                           property overides used by both the
+#                           drillbit (server) and client,
+#                           set in drill-env.sh or the environment.
+#   DRILLBIT_JAVA_OPTS      Optional JVM arguments specifically for
+#                           the server (drillbit). Set in the
+#                           environment or in the user defined
+#                           drill-env.sh
+#   SERVER_GC_OPTS          Defaults set in drill-config.sh, customized
+#                           in drill-env.sh.
+#   CP                      Drillbit classpath set in drill-config.sh
+#   args[]                  -Dname=value arguments to pass to the JVM
+#                           for per-run override of configuration options.
+
+cmd=$1
+shift
+
+drill_rotate_log ()
+{
+    log=$1;
+    num=5;
+    if [ -n "$2" ]; then
+    num=$2
+    fi
+    if [ -f "$log" ]; then # rotate logs
+    while [ $num -gt 1 ]; do
+        prev=`expr $num - 1`
+        [ -f "$log.$prev" ] && mv -f "$log.$prev" "$log.$num"
+        num=$prev
+    done
+    mv -f "$log" "$log.$num";
+    fi
+}
+
+# Enable GC logging if requested.
+# Note: if using YARN log dir, then no log rotation because each run under YARN
+# gets a new log directory.
+
+if [ "$SERVER_LOG_GC" == "1" ]; then
+  loggc="${DRILL_LOG_PREFIX}.gc"
+  SERVER_GC_OPTS="${SERVER_GC_OPTS} -Xloggc:${loggc}"
+  if [ -z "$DRILL_YARN_LOG_DIR" ]; then
+    drill_rotate_log $loggc
+  fi
+fi
+
+logqueries="${DRILL_LOG_PREFIX}_queries.json"
+LOG_OPTS="-Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$logqueries"
+if [ -n "$DRILL_JAVA_LIB_PATH" ]; then
+  DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS -Djava.library.path=$DRILL_JAVA_LIB_PATH"
+fi
+DRILL_ALL_JAVA_OPTS="$DRILLBIT_OPTS $DRILL_JAVA_OPTS $DRILLBIT_JAVA_OPTS $SERVER_GC_OPTS $@ $LOG_OPTS"
+BITCMD="$JAVA $DRILL_ALL_JAVA_OPTS -cp $CP org.apache.drill.exec.server.Drillbit"
+
+# The wrapper is purely for unit testing.
 
-. "$bin"/drill-config.sh
+if [ -n "$_DRILL_WRAPPER_" ]; then
+  BITCMD="$_DRILL_WRAPPER_ $BITCMD"
+fi
 
-DRILL_ALL_JAVA_OPTS="$DRILLBIT_JAVA_OPTS $DRILL_JAVA_OPTS $SERVER_GC_OPTS"
+# Run the command (exec) or just print it (debug).
+# Three options: run as a child (run), run & replace this process (exec) or
+# just print the command (debug).
 
-exec $JAVA -Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=$DRILLBIT_QUERY_LOG_PATH $DRILL_ALL_JAVA_OPTS -cp $CP org.apache.drill.exec.server.Drillbit
+case $cmd in
+(debug)
+  echo "----------------- Environment ------------------"
+  env
+  echo "------------------------------------------------"
+  echo "Launch command:"
+  echo $BITCMD
+  ;;
+(*)
+  exec $BITCMD
+  ;;
+esac

http://git-wip-us.apache.org/repos/asf/drill/blob/1f3c74da/distribution/src/resources/sqlline
----------------------------------------------------------------------
diff --git a/distribution/src/resources/sqlline b/distribution/src/resources/sqlline
index e494a72..dcdbd61 100644
--- a/distribution/src/resources/sqlline
+++ b/distribution/src/resources/sqlline
@@ -15,24 +15,40 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-ARGS=();
-while [ $# -gt 0 ] ; do
-  case "$1" in
-  -q) shift;
-      QUERY=$1;;
-  -e) shift;
-      QUERY=$1;;
-  -f) shift;
-      FILE=$1;;
-   *) ARGS+=($1);;
-  esac
-  shift
+# resolve links - "${BASH_SOURCE-$0}" may be a softlink
+this="${BASH_SOURCE-$0}"
+while [ -h "$this" ]; do
+  ls=`ls -ld "$this"`
+  link=`expr "$ls" : '.*-> \(.*\)$'`
+  if expr "$link" : '.*/.*' > /dev/null; then
+    this="$link"
+  else
+    this=`dirname "$this"`/"$link"
+  fi
 done
 
-bin=`dirname "${BASH_SOURCE-$0}"`
-bin=`cd "$bin">/dev/null; pwd`
+# convert relative path to absolute path
+bin=`dirname "$this"`
+
+# Do setup, including finding the site directory, removing the
+# --config argument, and copying remaining argument into the
+# args array.
+
+. "$bin/drill-config.sh"
 
-. "$bin"/drill-config.sh
+SLARGS=()
+for (( i=0; i < ${#args[@]}; i++ )); do
+  case "${args[i]}" in
+  -q|-e)
+      QUERY=$args[i+1]
+      let i=$i+1
+      ;;
+  -f) FILE=$args[i+1]
+      let i=$i+1
+      ;;
+   *) SLARGS+=("${args[i]}");;
+  esac
+done
 
 # Override SQLLine's default initial transaction isolation level.  (SQLLine
 # sets an initial level instead of leaving it at whatever the Driver's default
@@ -43,16 +59,32 @@ bin=`cd "$bin">/dev/null; pwd`
 # This is not currently needed as the new SQLLine we are using doesn't isolate.
 # DRILL_SHELL_JAVA_OPTS="-Dsqlline.isolation=TRANSACTION_NONE $DRILL_SHELL_JAVA_OPTS"
 
-DRILL_SHELL_JAVA_OPTS="$DRILL_SHELL_JAVA_OPTS -Dlog.path=$DRILL_LOG_DIR/sqlline.log -Dlog.query.path=$DRILL_LOG_DIR/sqlline_queries.json"
+DRILL_SHELL_LOG_OPTS="-Dlog.path=$DRILL_LOG_DIR/sqlline.log -Dlog.query.path=$DRILL_LOG_DIR/sqlline_queries.json"
+
+# Use either the SQLline options (for remote Drill) or full Drill options
+# (embedded Drill)
+
+if [ "$DRILL_EMBEDDED" = "1" ]; then
+  SQLLINE_JAVA_OPTS="$DRILL_JAVA_OPTS $DRILLBIT_OPTS"
+fi 
 
 if ! $is_cygwin; then
   DRILL_SHELL_OPTS="$DRILL_SHELL_OPTS --color=true"
 fi
 
+SHELL_OPTS="$DRILL_SHELL_JAVA_OPTS $SQLLINE_JAVA_OPTS $DRILL_SHELL_LOG_OPTS $CLIENT_GC_OPTS"
+CMD="$JAVA $SHELL_OPTS -cp $CP sqlline.SqlLine -d org.apache.drill.jdbc.Driver --maxWidth=10000"
+
+# The wrapper is purely for unit testing.
+
+if [ -n "$_DRILL_WRAPPER_" ]; then
+  CMD="$_DRILL_WRAPPER_ $CMD"
+fi
+
 if [ -n "$QUERY" ] ; then
-  echo $QUERY | exec "$JAVA" $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP sqlline.SqlLine -d org.apache.drill.jdbc.Driver  --maxWidth=10000 "${ARGS[@]}"
+  echo $QUERY | exec $CMD "${SLARGS[@]}"
 elif [ -n "$FILE" ] ; then
-  exec "$JAVA" $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP sqlline.SqlLine -d org.apache.drill.jdbc.Driver  --maxWidth=10000 "${ARGS[@]}" --run=$FILE
+  exec $CMD "${SLARGS[@]}" --run=$FILE
 else
-  exec "$JAVA" $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP sqlline.SqlLine -d org.apache.drill.jdbc.Driver --maxWidth=10000 $DRILL_SHELL_OPTS "${ARGS[@]}"
+  exec $CMD $DRILL_SHELL_OPTS "${SLARGS[@]}"
 fi


Mime
View raw message