flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [jira] [Commented] (FLINK-1679) Document how "degree of parallelism" / "parallelism" / "slots" are connected to each other
Date Thu, 12 Mar 2015 09:49:25 GMT
+1 for consistently calling it parallelism

-1 for AUTOMAX as the default

On Thu, Mar 12, 2015 at 10:31 AM, Robert Metzger <rmetzger@apache.org>
wrote:

> We can also make the change non-API breaking by adding an additional method
> and deprecating the old one.
>
>
> Why would the AUTOMAX parallelism eat up all cluster resources? It would
> only allocate all slots WITHIN the Flink cluster.
> Those users (=new users) who would benefit from the AUTOMAX parallelism
> have probably set the parallelism per TaskManager set to 1 anyways.
> Advanced users will set their parallelism / slots configuration anyways
> properly.
>
> In my experience, most users:
> - have exclusive access to a test cluster in the beginning (I don't think
> anybody who doesn't know the system at all would start Flink on a
> production cluster)
> - or use YARN
> - do not set any parallelism for jobs or slots per TaskManager.
>
> From these observations, I would actually set the number of slots on the
> TaskManagers to the number of available CPUs.
> And for the CLI frontend, I would by default let a job use all available
> slots (most users don't know that Flink allows to run multiple jobs at the
> same time).
>
> If users want to change the behavior, they have to look into the
> documentation.
>
> On Thu, Mar 12, 2015 at 10:20 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>
> > +1 for going consistently with parallelism. However, these are
> API-breaking
> > changes and we need to mark them deprecated before throwing them out,
> IMO.
> >
> > I am not comfortable with using AUTOMAX as a default. This is fine on
> > dedicated setups like YARN sessions, but will consume all available
> > resources of a cluster if a user forgets to set the -p flag (or fix the
> DOP
> > in the program). There is already a default-parallelsm flag in the config
> > and that value should be used, IMO.
> >
> > 2015-03-12 10:07 GMT+01:00 Robert Metzger (JIRA) <jira@apache.org>:
> >
> > >
> > >     [
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358345#comment-14358345
> > > ]
> > >
> > > Robert Metzger commented on FLINK-1679:
> > > ---------------------------------------
> > >
> > > I would suggest to remove all occurrences of "degreeOfParalleism" in
> the
> > > system and replace it by "parallelism" everywhere.
> > > The CLI frontend for example also calls it {{-p}}, not {{-dop}}.
> > >
> > > I would also suggest to set the parallelism by default to {{AUTOMAX}}
> in
> > > the CliFrontend.
> > >
> > > > Document how "degree of parallelism" /  "parallelism" / "slots" are
> > > connected to each other
> > > >
> > >
> >
> -------------------------------------------------------------------------------------------
> > > >
> > > >                 Key: FLINK-1679
> > > >                 URL:
> https://issues.apache.org/jira/browse/FLINK-1679
> > > >             Project: Flink
> > > >          Issue Type: Task
> > > >          Components: Documentation
> > > >    Affects Versions: 0.9
> > > >            Reporter: Robert Metzger
> > > >            Assignee: Ufuk Celebi
> > > >
> > > > I see too many users being confused about properly setting up Flink
> > with
> > > respect to parallelism.
> > >
> > >
> > >
> > > --
> > > This message was sent by Atlassian JIRA
> > > (v6.3.4#6332)
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message