flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From u..@apache.org
Subject flink git commit: [FLINK-4040] [dist] Allow JobManager/TaskManager-specific JVM args
Date Thu, 23 Jun 2016 08:17:21 GMT
Repository: flink
Updated Branches:
  refs/heads/master 479be148f -> b6e6b8189


[FLINK-4040] [dist] Allow JobManager/TaskManager-specific JVM args

- Adds two new config keys for the start up scripts:
    * env.java.opts.jobmanager
    * env.java.opts.taskmanager
- These are used in addition to the regular env.java.opts for the JobManager
  and TaskManager, respectively.
- Current behaviour is not changed. This does not address the JMX port
  setting, because that is handled differently.

This closes #2143.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/b6e6b818
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/b6e6b818
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/b6e6b818

Branch: refs/heads/master
Commit: b6e6b81896e56908294a3513f5ebd4857eff0d87
Parents: 479be14
Author: Ufuk Celebi <uce@apache.org>
Authored: Tue Jun 21 17:31:22 2016 +0200
Committer: Ufuk Celebi <uce@apache.org>
Committed: Thu Jun 23 10:16:54 2016 +0200

----------------------------------------------------------------------
 docs/setup/config.md                             |  6 +++++-
 flink-dist/src/main/flink-bin/bin/config.sh      | 16 ++++++++++++++++
 flink-dist/src/main/flink-bin/bin/jobmanager.sh  |  3 +++
 flink-dist/src/main/flink-bin/bin/taskmanager.sh |  5 ++++-
 4 files changed, 28 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/b6e6b818/docs/setup/config.md
----------------------------------------------------------------------
diff --git a/docs/setup/config.md b/docs/setup/config.md
index b4d0242..dde3e11 100644
--- a/docs/setup/config.md
+++ b/docs/setup/config.md
@@ -39,7 +39,11 @@ The configuration files for the TaskManagers can be different, Flink does
not as
 
 - `env.java.home`: The path to the Java installation to use (DEFAULT: system's default Java
installation, if found). Needs to be specified if the startup scripts fail to automatically
resolve the java home directory. Can be specified to point to a specific java installation
or version. If this option is not specified, the startup scripts also evaluate the `$JAVA_HOME`
environment variable.
 
-- `env.java.opts`: Set custom JVM options. This value is respected by Flink's start scripts
and Flink's YARN client. This can be used to set different garbage collectors or to include
remote debuggers into the JVMs running Flink's services.
+- `env.java.opts`: Set custom JVM options. This value is respected by Flink's start scripts,
both JobManager and TaskManager, and Flink's YARN client. This can be used to set different
garbage collectors or to include remote debuggers into the JVMs running Flink's services.
Use `env.java.opts.jobmanager` and `env.java.opts.taskmanager` for JobManager or TaskManager-specific
options, respectively.
+
+- `env.java.opts.jobmanager`: JobManager-specific JVM options. These are used in addition
to the regular `env.java.opts`.
+
+- `env.java.opts.taskmanager`: TaskManager-specific JVM options. These are used in addition
to the regular `env.java.opts`.
 
 - `jobmanager.rpc.address`: The IP address of the JobManager, which is the master/coordinator
of the distributed system (DEFAULT: localhost).
 

http://git-wip-us.apache.org/repos/asf/flink/blob/b6e6b818/flink-dist/src/main/flink-bin/bin/config.sh
----------------------------------------------------------------------
diff --git a/flink-dist/src/main/flink-bin/bin/config.sh b/flink-dist/src/main/flink-bin/bin/config.sh
index b6bdbed..96b3122 100755
--- a/flink-dist/src/main/flink-bin/bin/config.sh
+++ b/flink-dist/src/main/flink-bin/bin/config.sh
@@ -81,6 +81,8 @@ readFromConfig() {
 DEFAULT_ENV_PID_DIR="/tmp"                          # Directory to store *.pid files to
 DEFAULT_ENV_LOG_MAX=5                               # Maximum number of old log files to
keep
 DEFAULT_ENV_JAVA_OPTS=""                            # Optional JVM args
+DEFAULT_ENV_JAVA_OPTS_JM=""                         # Optional JVM args (JobManager)
+DEFAULT_ENV_JAVA_OPTS_TM=""                         # Optional JVM args (TaskManager)
 DEFAULT_ENV_SSH_OPTS=""                             # Optional SSH parameters running in
cluster mode
 
 ########################################################################################################################
@@ -99,6 +101,8 @@ KEY_ENV_LOG_DIR="env.log.dir"
 KEY_ENV_LOG_MAX="env.log.max"
 KEY_ENV_JAVA_HOME="env.java.home"
 KEY_ENV_JAVA_OPTS="env.java.opts"
+KEY_ENV_JAVA_OPTS_JM="env.java.opts.jobmanager"
+KEY_ENV_JAVA_OPTS_TM="env.java.opts.taskmanager"
 KEY_ENV_SSH_OPTS="env.ssh.opts"
 KEY_RECOVERY_MODE="recovery.mode"
 KEY_ZK_HEAP_MB="zookeeper.heap.mb"
@@ -229,6 +233,18 @@ if [ -z "${FLINK_ENV_JAVA_OPTS}" ]; then
     FLINK_ENV_JAVA_OPTS="$( echo "${FLINK_ENV_JAVA_OPTS}" | sed -e 's/^"//'  -e 's/"$//'
)"
 fi
 
+if [ -z "${FLINK_ENV_JAVA_OPTS_JM}" ]; then
+    FLINK_ENV_JAVA_OPTS_JM=$(readFromConfig ${KEY_ENV_JAVA_OPTS_JM} "${DEFAULT_ENV_JAVA_OPTS_JM}"
"${YAML_CONF}")
+    # Remove leading and ending double quotes (if present) of value
+    FLINK_ENV_JAVA_OPTS_JM="$( echo "${FLINK_ENV_JAVA_OPTS_JM}" | sed -e 's/^"//'  -e 's/"$//'
)"
+fi
+
+if [ -z "${FLINK_ENV_JAVA_OPTS_TM}" ]; then
+    FLINK_ENV_JAVA_OPTS_TM=$(readFromConfig ${KEY_ENV_JAVA_OPTS_TM} "${DEFAULT_ENV_JAVA_OPTS_TM}"
"${YAML_CONF}")
+    # Remove leading and ending double quotes (if present) of value
+    FLINK_ENV_JAVA_OPTS_TM="$( echo "${FLINK_ENV_JAVA_OPTS_TM}" | sed -e 's/^"//'  -e 's/"$//'
)"
+fi
+
 if [ -z "${FLINK_SSH_OPTS}" ]; then
     FLINK_SSH_OPTS=$(readFromConfig ${KEY_ENV_SSH_OPTS} "${DEFAULT_ENV_SSH_OPTS}" "${YAML_CONF}")
 fi

http://git-wip-us.apache.org/repos/asf/flink/blob/b6e6b818/flink-dist/src/main/flink-bin/bin/jobmanager.sh
----------------------------------------------------------------------
diff --git a/flink-dist/src/main/flink-bin/bin/jobmanager.sh b/flink-dist/src/main/flink-bin/bin/jobmanager.sh
index 9ec90ce..5610290 100755
--- a/flink-dist/src/main/flink-bin/bin/jobmanager.sh
+++ b/flink-dist/src/main/flink-bin/bin/jobmanager.sh
@@ -54,6 +54,9 @@ if [[ $STARTSTOP == "start" ]]; then
         export JVM_ARGS="$JVM_ARGS -Xms"$FLINK_JM_HEAP"m -Xmx"$FLINK_JM_HEAP"m"
     fi
 
+    # Add JobManager-specific JVM options
+    export FLINK_ENV_JAVA_OPTS="${FLINK_ENV_JAVA_OPTS} ${FLINK_ENV_JAVA_OPTS_JM}"
+
     # Startup parameters
     args=("--configDir" "${FLINK_CONF_DIR}" "--executionMode" "${EXECUTIONMODE}")
     if [ ! -z $HOST ]; then

http://git-wip-us.apache.org/repos/asf/flink/blob/b6e6b818/flink-dist/src/main/flink-bin/bin/taskmanager.sh
----------------------------------------------------------------------
diff --git a/flink-dist/src/main/flink-bin/bin/taskmanager.sh b/flink-dist/src/main/flink-bin/bin/taskmanager.sh
index 62847be..e579c0c 100755
--- a/flink-dist/src/main/flink-bin/bin/taskmanager.sh
+++ b/flink-dist/src/main/flink-bin/bin/taskmanager.sh
@@ -31,7 +31,7 @@ if [[ $STARTSTOP == "start" ]]; then
 
     # if memory allocation mode is lazy and no other JVM options are set,
     # set the 'Concurrent Mark Sweep GC'
-    if [[ $FLINK_TM_MEM_PRE_ALLOCATE == "false" ]] && [ -z "${FLINK_ENV_JAVA_OPTS}"
]; then
+    if [[ $FLINK_TM_MEM_PRE_ALLOCATE == "false" ]] && [ -z "${FLINK_ENV_JAVA_OPTS}"
] && [ -z "${FLINK_ENV_JAVA_OPTS_TM}" ]; then
 
         JAVA_VERSION=$($JAVA_RUN -version 2>&1 | sed 's/.*version "\(.*\)\.\(.*\)\..*"/\1\2/;
1q')
 
@@ -89,6 +89,9 @@ if [[ $STARTSTOP == "start" ]]; then
 
     fi
 
+    # Add TaskManager-specific JVM options
+    export FLINK_ENV_JAVA_OPTS="${FLINK_ENV_JAVA_OPTS} ${FLINK_ENV_JAVA_OPTS_TM}"
+
     # Startup parameters
     args=("--configDir" "${FLINK_CONF_DIR}")
 fi


Mime
View raw message