flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Distributing Tasks over Task manager
Date Wed, 12 Oct 2016 15:43:51 GMT
Hi Jürgen,

Are you using the DataStream or the DataSet API?
Maybe the operator chaining is causing too many operations to be "packed"
into one task. Check out this documentation page:
https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#task-chaining-and-resource-groups

You could try to disable chaining completely to see if that resolves the
issue (you'll probably pay for this by having more serialization overhead
and network traffic).

If my suggestions don't help, can you post a screenshot of your job plan
(from the web interface) here, so that we see what operations you are
performing?

Regards,
Robert



On Wed, Oct 12, 2016 at 12:52 PM, Jürgen Thomann <
juergen.thomann@innogames.com> wrote:

> Hi,
>
> we currently have an issue with Flink, as it allocates many tasks to the
> same task manager and as a result it overloads it. I reduced the amount of
> task slots per task manager (keeping the CPU count) and added some more
> servers but that did not help to distribute the load.
>
> Is there some way to force Flink to distribute the load/tasks on a
> standalone cluster? I saw that https://issues.apache.org/jira
> /browse/FLINK-1003 would maybe provide what we need, but that is
> currently not worked on as it seems.
>
> Cheers,
> Jürgen
>

Mime
View raw message