flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: How to configure max parallelism?
Date Wed, 25 Jan 2017 10:10:35 GMT
Hey Gyula,

as far as I can tell, there are no docs for this yet. Good news is
that the docs are getting a lot of love for the release and I guess
this is simply something that is still missing.

The setMaxParallelism call should be available in all the places where
you can also call setParallelism. You have the option to set it on the
environment and per operator.

If you don't configure anything, we compute somewhat sane defaults in
order to allow scaling up later:
Parallelism 1-64 => default max parallelism: 128
Parallelism 65+ => parallelism + parallelism/2 (where I parallelism is
rounded up to the next power of two)

The maximum allowed max parallelism is 32768.

The computation happens here:
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyGroupRangeAssignment.java#L126.

– Ufuk


On Wed, Jan 25, 2017 at 9:05 AM, Gyula Fóra <gyula.fora@gmail.com> wrote:
> Hi all,
>
> I can't seem to find in the documentation how to set the maximum parallelism
> for rescaling keyed state.
>
> Can anyone help me out here?
>
> Thanks!
> Gyula

Mime
View raw message