flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isha Arkatkar <icarkat...@gmail.com>
Subject Re: Question on Task Chaining
Date Sun, 08 Nov 2015 04:53:22 GMT
Thanks Stephan! That helps :)

Regards,
Isha

On Sat, Nov 7, 2015 at 7:08 AM, Stephan Ewen <sewen@apache.org> wrote:

> Hi!
>
> Chained tasks run in the same thread, and there is no serialization
> involved. The FilterFunction is directly called on the result of the
> MapFunction.
>
> Records between non-chained tasks always go through the serialization
> stack.
>
> Even in the case of non-chained tasks, the different operators share a slot
> in the TaskManager, which means they are executed in the same JVM. If you
> want to be sure, check the web dashboard. You can click in the map and on
> the filter operator to check where all the subtasks run. If the subtask (n)
> from the map and filer function have the same host/port for the
> TaskManager, then they are in the same JVM.
>
> To prevent sharing a slot, use the "startNewResourceGroup()" call. In that
> case, the FilterFunction will be in a different slot than the MapFunction.
> The system will try to co-locate them (locality aware scheduling), so they
> may end up in the same JVM if that TaskManager had more free slots. If not,
> they run in different JVMs.
>
> Hope that helps...
>
> Greetings,
> Stephan
>
> On Sat, Nov 7, 2015 at 2:26 AM, Isha Arkatkar <icarkatkar@gmail.com>
> wrote:
>
> > Hi all,
> >
> >    I am new to flink and wanted to understand effects of setting task
> > chaining and resource group allocation. I tried following configurations,
> > with task slots set to 2:
> >
> > 1. map(..).disableChaining.filter(..)
> >      In this case, I can see 2 vertices for this application, both on the
> > same node. Do  these two operators share JVM? If not, does it go through
> > serialization flow?
> >
> > 2. If I do not specify disableChaining, does that mean the map and filter
> > operations run on the same thread or on the same JVM?
> >
> > Thanks!
> > Isha
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message