flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Warneke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-937) Change the YARN Client to allocate all cluster resources, if no argument given
Date Mon, 16 Jun 2014 22:02:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033072#comment-14033072

Daniel Warneke commented on FLINK-937:

To be honest, I am not a big fan of interactive tools. Couldn't we use the PACT compiler to
derive reasonable defaults for the number and type of YARN containers?

In the long run, in order to make Flink run more smoothly on YARN, I think it is crucial to
move to a processing model that does not require all resources to be available simultaneously.
Correct me if I'm wrong, but YARN does not support gang scheduling at moment, so greadily
hogging containers until a job is ready to run can easily result in deadlocks in a multi-tenant

> Change the YARN Client to allocate all cluster resources, if no argument given
> ------------------------------------------------------------------------------
>                 Key: FLINK-937
>                 URL: https://issues.apache.org/jira/browse/FLINK-937
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN Client
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
>             Fix For: 0.6-incubating
> In order to further improve the user experience, I would like to change the YARN client's
behavior to allocate as many cluster resources as possible, if the user does not specify differently.
> The majority of users have exclusive access to the cluster.

This message was sent by Atlassian JIRA

View raw message