flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9969) Unreasonable memory requirements to complete examples/batch/WordCount
Date Fri, 27 Jul 2018 11:09:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559576#comment-16559576
] 

Piotr Nowojski commented on FLINK-9969:
---------------------------------------

Strangely, problem disappears when {{taskmanager.memory.preallocate}} is set to true

> Unreasonable memory requirements to complete examples/batch/WordCount
> ---------------------------------------------------------------------
>
>                 Key: FLINK-9969
>                 URL: https://issues.apache.org/jira/browse/FLINK-9969
>             Project: Flink
>          Issue Type: Bug
>          Components: ResourceManager
>    Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.6.0
>            Reporter: Piotr Nowojski
>            Priority: Blocker
>             Fix For: 1.5.2, 1.6.0
>
>
> setup on AWS EMR:
>  * 5 worker nodes (m4.4xlarge nodes) 
>  * 1 master node (m4.large)
> following command fails with out of memory errors:
> {noformat}
> export HADOOP_CLASSPATH=`hadoop classpath`
> ./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 16000 examples/batch/WordCount.jar{noformat}
> Only increasing memory over 17.2GB example completes. At the same time after disabling
flip6 following command succeeds:
> {noformat}
> export HADOOP_CLASSPATH=`hadoop classpath`
> ./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 1000 examples/batch/WordCount.jar{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message