flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Warneke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-937) Change the YARN Client to allocate all cluster resources, if no argument given
Date Tue, 17 Jun 2014 21:23:03 GMT

    [ https://issues.apache.org/jira/browse/FLINK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034389#comment-14034389

Daniel Warneke commented on FLINK-937:

As far as I know, queues are rather an operational feature cluster administrators can use
to handle different job priorities or isolate different types of workloads. I don’t see
how they could help to find a reasonable default degree of parallelism. To the best of my
knowledge, the only YARN APIs we can rely on are:

- Min/max size of containers
- Total number of nodes in the cluster (idle or not)

Therefore, I basically see two possibilities for the default behavior:

- Start large: Determine number of nodes and use them all (Robert’s initial suggestions?)
- Start small: Make default degree of parallelism 1 or 2 (default behavior of MapReduce)

I have a slight preference for option 2, but I would choose either one over the “assistant”…

> Change the YARN Client to allocate all cluster resources, if no argument given
> ------------------------------------------------------------------------------
>                 Key: FLINK-937
>                 URL: https://issues.apache.org/jira/browse/FLINK-937
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN Client
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
>             Fix For: 0.6-incubating
> In order to further improve the user experience, I would like to change the YARN client's
behavior to allocate as many cluster resources as possible, if the user does not specify differently.
> The majority of users have exclusive access to the cluster.

This message was sent by Atlassian JIRA

View raw message