flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saliya Ekanayake <esal...@gmail.com>
Subject Re: Parameters to Control Intra-node Parallelism
Date Sat, 02 Jul 2016 15:12:47 GMT
That's great, so is there support to pin task managers to sockets as well?

On Sat, Jul 2, 2016 at 11:08 AM, Ufuk Celebi <uce@apache.org> wrote:

> Regarding 2) if you don't manually configure something else, that
> should happen always.
>
> Yes, you can run more than one task manager per node depending on the
> process isolation you want. Within a task manager, there are multiple
> threads for each slot. For example, if you have 2 task managers with 2
> slots each and submit a job with parallelism 4, each task manager will
> execute 2 sub tasks in separate Threads.
>
>
> On Sat, Jul 2, 2016 at 3:26 AM, Saliya Ekanayake <esaliya@gmail.com>
> wrote:
> > Hi Ufuk,
> >
> > Looking at the document you sent it seems only 1 task manager per node
> exist
> > and within that you have multiple slots. Is it possible to run more than
> 1
> > task manager per node? Also, within a task manager is the parallelism
> done
> > through threads or processes?
> >
> > Thank you,
> > Saliya
> >
> > On Thu, Jun 30, 2016 at 5:27 PM, Saliya Ekanayake <esaliya@gmail.com>
> wrote:
> >>
> >> Thank you, I'll check these.
> >>
> >> In 2.) you said they are likely to exchange through memory. Is there a
> >> case why they wouldn't?
> >>
> >> On Thu, Jun 30, 2016 at 5:03 AM, Ufuk Celebi <uce@apache.org> wrote:
> >>>
> >>> On Thu, Jun 30, 2016 at 1:44 AM, Saliya Ekanayake <esaliya@gmail.com>
> >>> wrote:
> >>> > 1. What parameters are available to control parallelism within a
> node?
> >>>
> >>> Task Manager processing slots:
> >>>
> >>>
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-taskmanager-processing-slots
> >>>
> >>> > 2. Does Flink support shared memory-based messaging within a node
> >>> > (without
> >>> > doing TCP calls)?
> >>>
> >>> Yes, local exchanges happen via memory and not TCP, for example if you
> >>> have a map-reduce, map subtask 1 and reduce subtask 1 are likely to
> >>> exchange data locally.
> >>>
> >>> > 3. Is there support for Infiniband interconnect?
> >>>
> >>> No, not that I'm aware of.
> >>>
> >>> – Ufuk
> >>
> >>
> >>
> >>
> >> --
> >> Saliya Ekanayake
> >> Ph.D. Candidate | Research Assistant
> >> School of Informatics and Computing | Digital Science Center
> >> Indiana University, Bloomington
> >>
> >
> >
> >
> > --
> > Saliya Ekanayake
> > Ph.D. Candidate | Research Assistant
> > School of Informatics and Computing | Digital Science Center
> > Indiana University, Bloomington
> >
>



-- 
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington

Mime
View raw message