flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Parameters to Control Intra-node Parallelism
Date Sun, 03 Jul 2016 15:28:37 GMT
Yes, exactly.

On Sat, Jul 2, 2016 at 6:28 PM, Saliya Ekanayake <esaliya@gmail.com> wrote:
> Thank you, yes, it can be done externally, if not supported within Flink.
>
> So the way to spawn multiple task managers would be to list the same slave
> machines N times as necessary in the slaves file?
>
> On Sat, Jul 2, 2016 at 11:22 AM, Ufuk Celebi <uce@apache.org> wrote:
>>
>> No, not inside of Flink. That sounds like something like the OS or
>> resource manager should handle.
>>
>> On Sat, Jul 2, 2016 at 5:12 PM, Saliya Ekanayake <esaliya@gmail.com>
>> wrote:
>> > That's great, so is there support to pin task managers to sockets as
>> > well?
>> >
>> > On Sat, Jul 2, 2016 at 11:08 AM, Ufuk Celebi <uce@apache.org> wrote:
>> >>
>> >> Regarding 2) if you don't manually configure something else, that
>> >> should happen always.
>> >>
>> >> Yes, you can run more than one task manager per node depending on the
>> >> process isolation you want. Within a task manager, there are multiple
>> >> threads for each slot. For example, if you have 2 task managers with 2
>> >> slots each and submit a job with parallelism 4, each task manager will
>> >> execute 2 sub tasks in separate Threads.
>> >>
>> >>
>> >> On Sat, Jul 2, 2016 at 3:26 AM, Saliya Ekanayake <esaliya@gmail.com>
>> >> wrote:
>> >> > Hi Ufuk,
>> >> >
>> >> > Looking at the document you sent it seems only 1 task manager per
>> >> > node
>> >> > exist
>> >> > and within that you have multiple slots. Is it possible to run more
>> >> > than
>> >> > 1
>> >> > task manager per node? Also, within a task manager is the parallelism
>> >> > done
>> >> > through threads or processes?
>> >> >
>> >> > Thank you,
>> >> > Saliya
>> >> >
>> >> > On Thu, Jun 30, 2016 at 5:27 PM, Saliya Ekanayake <esaliya@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Thank you, I'll check these.
>> >> >>
>> >> >> In 2.) you said they are likely to exchange through memory. Is
there
>> >> >> a
>> >> >> case why they wouldn't?
>> >> >>
>> >> >> On Thu, Jun 30, 2016 at 5:03 AM, Ufuk Celebi <uce@apache.org>
wrote:
>> >> >>>
>> >> >>> On Thu, Jun 30, 2016 at 1:44 AM, Saliya Ekanayake
>> >> >>> <esaliya@gmail.com>
>> >> >>> wrote:
>> >> >>> > 1. What parameters are available to control parallelism
within a
>> >> >>> > node?
>> >> >>>
>> >> >>> Task Manager processing slots:
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-taskmanager-processing-slots
>> >> >>>
>> >> >>> > 2. Does Flink support shared memory-based messaging within
a node
>> >> >>> > (without
>> >> >>> > doing TCP calls)?
>> >> >>>
>> >> >>> Yes, local exchanges happen via memory and not TCP, for example
if
>> >> >>> you
>> >> >>> have a map-reduce, map subtask 1 and reduce subtask 1 are likely
to
>> >> >>> exchange data locally.
>> >> >>>
>> >> >>> > 3. Is there support for Infiniband interconnect?
>> >> >>>
>> >> >>> No, not that I'm aware of.
>> >> >>>
>> >> >>> – Ufuk
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Saliya Ekanayake
>> >> >> Ph.D. Candidate | Research Assistant
>> >> >> School of Informatics and Computing | Digital Science Center
>> >> >> Indiana University, Bloomington
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Saliya Ekanayake
>> >> > Ph.D. Candidate | Research Assistant
>> >> > School of Informatics and Computing | Digital Science Center
>> >> > Indiana University, Bloomington
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Saliya Ekanayake
>> > Ph.D. Candidate | Research Assistant
>> > School of Informatics and Computing | Digital Science Center
>> > Indiana University, Bloomington
>> >
>
>
>
>
> --
> Saliya Ekanayake
> Ph.D. Candidate | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
>

Mime
View raw message