flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhupesh Chawda <bhup...@datatorrent.com>
Subject Re: Scheduling of Flink jobs
Date Thu, 01 Sep 2016 11:27:52 GMT
Thanks Stephan for your reply.

If I understand correctly, if my parallelism is 1, then all of the
operators, not matter how many (say 20), will still run on just one task
manager.
What happens in case the resources on that task manager are not sufficient
for all of these operators?

~ Bhupesh


On Thu, Sep 1, 2016 at 3:32 PM, Stephan Ewen <sewen@apache.org> wrote:

> In the default configuration, the job uses as many slots as the parallelism
> of the operators states. I assume you run with a parallelism of 2, so it
> occupies two slots.
>
> if you run 5 taskmanagers with each one slot, you should set the
> parallelism to 5 as well.
>
> On Mon, Aug 29, 2016 at 4:04 PM, Bhupesh Chawda <bhupesh@apache.org>
> wrote:
>
> > Hi,
> >
> > I am running Flink on a cluster of 5 nodes.
> > Here is my config:
> >
> >
> >
> > *taskmanager.numberOfTaskSlots: 1parallelism.default: 1*
> > My Flink dashboard shows the following:
> >
> > *Task Managers: 5*
> >
> > *Task Slots: 5*
> >
> > *Available Task Slots: 5*
> > I have the following questions:
> >
> >    1. Why does a job with 8 tasks occupy only 2 task slots (3 slots
> remain
> >    free as seen from the UI)? As per my understanding, since the number
> of
> >    Task Slots as shown above is just 5, perhaps this job may not get
> enough
> >    resources (task slots).
> >    2. I notice that most of the tasks (operators) in the job run on just
> >    one of the nodes. The other nodes are idle and free. Is there any way
> to
> >    distribute the tasks among other nodes more evenly?
> >
> > Please advice.
> >
> > Thanks.
> >
> > ~ Bhupesh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message