flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2641) Integrate the off-heap memory configuration with the TaskManager start script
Date Tue, 15 Sep 2015 09:10:46 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745098#comment-14745098
] 

ASF GitHub Bot commented on FLINK-2641:
---------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1129#discussion_r39488999
  
    --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
    @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then
             fi
         fi
     
    -    if [[ ! ${FLINK_TM_HEAP} =~ ${IS_NUMBER} ]]; then
    -        echo "[ERROR] Configured TaskManager JVM heap size is not a number. Please set
'$KEY_TASKM_HEAP_MB' in $FLINK_CONF_FILE."
    +    if [[ ! ${FLINK_TM_HEAP} =~ ${IS_NUMBER} ]] || [[ "${FLINK_TM_HEAP}" -lt "0" ]];
then
    +        echo "[ERROR] Configured TaskManager JVM heap size is not a number. Please set
'${KEY_TASKM_MEM_SIZE}' in ${FLINK_CONF_FILE}."
             exit 1
         fi
     
    -    if [ "$FLINK_TM_HEAP" -gt 0 ]; then
    -        export JVM_ARGS="$JVM_ARGS -Xms"$FLINK_TM_HEAP"m -Xmx"$FLINK_TM_HEAP"m"
    +    if [ "${FLINK_TM_HEAP}" -gt "0" ]; then
    +
    +        TM_HEAP_SIZE=${FLINK_TM_HEAP}
    +        TM_OFFHEAP_SIZE=0
    +        # some space for Netty initilization
    +        NETTY_BUFFERS=$((1024 * 1024))
    +
    +        if useOffHeapMemory; then
    +            if [[ "${FLINK_TM_MEM_MANAGED_SIZE}" -gt "0" ]]; then
    +                # We split up the total memory in heap and off-heap memory
    +                if [[ "${FLINK_TM_HEAP}" -le "${FLINK_TM_MEM_MANAGED_SIZE}" ]]; then
    +                    echo "[ERROR] Configured TaskManager memory size ('${KEY_TASKM_MEM_SIZE}')
must be larger than the managed memory size ('${KEY_TASKM_MEM_MANAGED_SIZE}')."
    +                    exit 1
    +                fi
    +                TM_OFFHEAP_SIZE=${FLINK_TM_MEM_MANAGED_SIZE}
    +                TM_HEAP_SIZE=$((FLINK_TM_HEAP - FLINK_TM_MEM_MANAGED_SIZE))
    +            else
    +                # We calculate the memory using a fraction of the total memory
    +                if [[ `bc -l <<< "${FLINK_TM_MEM_MANAGED_FRACTION} >= 1.0"`
!= "0" ]]; then
    +                    echo "[ERROR] Configured TaskManager managed memory fraction is not
a valid value. Please set '${KEY_TASKM_MEM_MANAGED_FRACTION}' in ${FLINK_CONF_FILE}"
    +                    exit 1
    +                fi
    +                # recalculate the JVM heap memory by taking the off-heap ratio into account
    +                TM_OFFHEAP_SIZE=`printf '%.0f\n' $(bc -l <<< "${FLINK_TM_HEAP}
* ${FLINK_TM_MEM_MANAGED_FRACTION}")`
    +                TM_HEAP_SIZE=$((FLINK_TM_HEAP - TM_OFFHEAP_SIZE))
    +            fi
    +        fi
    +
    +        export JVM_ARGS="${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M -XX:MaxDirectMemorySize=$((TM_OFFHEAP_SIZE
* 1024 * 1024 + FLINK_TM_MEM_NETWORK_SIZE + NETTY_BUFFERS))"
    --- End diff --
    
    Append "M" to -XX:MaxDirectMemorySize parameter?


> Integrate the off-heap memory configuration with the TaskManager start script
> -----------------------------------------------------------------------------
>
>                 Key: FLINK-2641
>                 URL: https://issues.apache.org/jira/browse/FLINK-2641
>             Project: Flink
>          Issue Type: New Feature
>          Components: Start-Stop Scripts
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Maximilian Michels
>             Fix For: 0.10
>
>
> The TaskManager start script needs to adjust the {{-Xmx}}, {{-Xms}}, and {{-XX:MaxDirectMemorySize}}
parameters according to the off-heap memory settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message