flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-8826) In Flip6 mode, when starting yarn cluster, configured taskmanager.heap.mb is ignored
Date Thu, 01 Mar 2018 16:14:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-8826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Piotr Nowojski updated FLINK-8826:
----------------------------------
    Description: 
When I tried running some job on the cluster, despite setting 

taskmanager.heap.mb = 3072

taskmanager.network.memory.fraction: 0.4

and reported in the console
{code:java}
Cluster specification: ClusterSpecification{masterMemoryMB=768, taskManagerMemoryMB=3072,
numberTaskManagers=92, slotsPerTaskManager=1}{code}
The actual settings were:
{noformat}
 

2018-03-01 14:53:18,918 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        - --------------------------------------------------------------------------------

2018-03-01 14:53:18,921 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018
@ 17:43:39 UTC)

2018-03-01 14:53:18,921 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  OS current user: yarn

2018-03-01 14:53:19,780 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Current Hadoop/Kerberos user: hadoop

2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14

2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Maximum heap size: 245 MiBytes

2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  JAVA_HOME: /usr/lib/jvm/java-openjdk

2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Hadoop version: 2.4.1

2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  JVM Options:

2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Xms255m

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Xmx255m

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -XX:MaxDirectMemorySize=769m

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Dlogback.configurationFile=file:./logback.xml

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Dlog4j.configuration=file:./log4j.properties

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Program Arguments:

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     --configDir{noformat}
Heap was set to 255, while with default cuts of it should be 1383. 255MB seems like coming
from default taskmanager.heap.mb value of 1024.

When starting in non flip6 everything works as expected:
{noformat}
 

2018-03-01 14:04:49,650 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      - --------------------------------------------------------------------------------

2018-03-01 14:04:49,700 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Starting YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018
@ 17:43:39 UTC)

2018-03-01 14:04:49,700 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  OS current user: yarn

2018-03-01 14:04:53,277 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Current Hadoop/Kerberos user: hadoop

2018-03-01 14:04:53,278 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14

2018-03-01 14:04:53,279 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Maximum heap size: 1326 MiBytes

2018-03-01 14:04:53,279 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  JAVA_HOME: /usr/lib/jvm/java-openjdk

2018-03-01 14:04:53,282 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Hadoop version: 2.4.1

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  JVM Options:

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Xms1383m

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Xmx1383m

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -XX:MaxDirectMemorySize=1689m

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log

2018-03-01 14:04:53,285 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Dlogback.configurationFile=file:./logback.xml

2018-03-01 14:04:53,286 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Dlog4j.configuration=file:./log4j.properties

2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Program Arguments:

2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     --configDir

2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     .{noformat}
 

CC [~till.rohrmann]

  was:
When I tried running some job on the cluster, despite setting 

taskmanager.heap.mb = 3072

taskmanager.network.memory.fraction: 0.4

and reported in the console

{{

Cluster specification: ClusterSpecification\{masterMemoryMB=768, taskManagerMemoryMB=3072,
numberTaskManagers=92, slotsPerTaskManager=1}

}}

The actual settings were:

{{

2018-03-01 14:53:18,918 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        - --------------------------------------------------------------------------------

2018-03-01 14:53:18,921 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018
@ 17:43:39 UTC)

2018-03-01 14:53:18,921 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  OS current user: yarn

2018-03-01 14:53:19,780 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Current Hadoop/Kerberos user: hadoop

2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14

2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Maximum heap size: 245 MiBytes

2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  JAVA_HOME: /usr/lib/jvm/java-openjdk

2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Hadoop version: 2.4.1

2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  JVM Options:

2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Xms255m

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Xmx255m

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -XX:MaxDirectMemorySize=769m

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Dlogback.configurationFile=file:./logback.xml

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     -Dlog4j.configuration=file:./log4j.properties

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -  Program Arguments:

2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner         
        -     --configDir

}}

Heap was set to 255, while with default cuts of it should be 1383. 255MB seems like coming
from default taskmanager.heap.mb value of 1024.

 

When starting in non flip6 everything works as expected:

{{

2018-03-01 14:04:49,650 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      - --------------------------------------------------------------------------------

2018-03-01 14:04:49,700 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Starting YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018
@ 17:43:39 UTC)

2018-03-01 14:04:49,700 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  OS current user: yarn

2018-03-01 14:04:53,277 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Current Hadoop/Kerberos user: hadoop

2018-03-01 14:04:53,278 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14

2018-03-01 14:04:53,279 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Maximum heap size: 1326 MiBytes

2018-03-01 14:04:53,279 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  JAVA_HOME: /usr/lib/jvm/java-openjdk

2018-03-01 14:04:53,282 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Hadoop version: 2.4.1

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  JVM Options:

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Xms1383m

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Xmx1383m

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -XX:MaxDirectMemorySize=1689m

2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log

2018-03-01 14:04:53,285 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Dlogback.configurationFile=file:./logback.xml

2018-03-01 14:04:53,286 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     -Dlog4j.configuration=file:./log4j.properties

2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -  Program Arguments:

2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     --configDir

2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory     
      -     .

}}

 

CC [~till.rohrmann]


> In Flip6 mode, when starting yarn cluster, configured taskmanager.heap.mb is ignored
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-8826
>                 URL: https://issues.apache.org/jira/browse/FLINK-8826
>             Project: Flink
>          Issue Type: Bug
>          Components: ResourceManager, YARN
>    Affects Versions: 1.5.0
>            Reporter: Piotr Nowojski
>            Priority: Blocker
>
> When I tried running some job on the cluster, despite setting 
> taskmanager.heap.mb = 3072
> taskmanager.network.memory.fraction: 0.4
> and reported in the console
> {code:java}
> Cluster specification: ClusterSpecification{masterMemoryMB=768, taskManagerMemoryMB=3072,
numberTaskManagers=92, slotsPerTaskManager=1}{code}
> The actual settings were:
> {noformat}
>  
> 2018-03-01 14:53:18,918 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          - --------------------------------------------------------------------------------
> 2018-03-01 14:53:18,921 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT, Rev:e92eb39,
Date:28.02.2018 @ 17:43:39 UTC)
> 2018-03-01 14:53:18,921 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  OS current user: yarn
> 2018-03-01 14:53:19,780 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  Current Hadoop/Kerberos user: hadoop
> 2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  Maximum heap size: 245 MiBytes
> 2018-03-01 14:53:19,781 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  Hadoop version: 2.4.1
> 2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  JVM Options:
> 2018-03-01 14:53:19,783 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     -Xms255m
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     -Xmx255m
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     -XX:MaxDirectMemorySize=769m
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     -Dlogback.configurationFile=file:./logback.xml
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     -Dlog4j.configuration=file:./log4j.properties
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -  Program Arguments:
> 2018-03-01 14:53:19,784 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner       
          -     --configDir{noformat}
> Heap was set to 255, while with default cuts of it should be 1383. 255MB seems like coming
from default taskmanager.heap.mb value of 1024.
> When starting in non flip6 everything works as expected:
> {noformat}
>  
> 2018-03-01 14:04:49,650 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        - --------------------------------------------------------------------------------
> 2018-03-01 14:04:49,700 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  Starting YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018
@ 17:43:39 UTC)
> 2018-03-01 14:04:49,700 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  OS current user: yarn
> 2018-03-01 14:04:53,277 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  Current Hadoop/Kerberos user: hadoop
> 2018-03-01 14:04:53,278 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-03-01 14:04:53,279 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  Maximum heap size: 1326 MiBytes
> 2018-03-01 14:04:53,279 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-03-01 14:04:53,282 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  Hadoop version: 2.4.1
> 2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  JVM Options:
> 2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     -Xms1383m
> 2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     -Xmx1383m
> 2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     -XX:MaxDirectMemorySize=1689m
> 2018-03-01 14:04:53,284 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
> 2018-03-01 14:04:53,285 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     -Dlogback.configurationFile=file:./logback.xml
> 2018-03-01 14:04:53,286 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     -Dlog4j.configuration=file:./log4j.properties
> 2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -  Program Arguments:
> 2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     --configDir
> 2018-03-01 14:04:53,287 INFO  org.apache.flink.yarn.YarnTaskManagerRunnerFactory   
        -     .{noformat}
>  
> CC [~till.rohrmann]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message