flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Parameters to Control Intra-node Parallelism
Date Thu, 07 Jul 2016 08:27:36 GMT
No that should suffice. Can you check whether there are any task
manager logs for the second TM on that machine
(taskmanager-X-j-011.log where X is the TM number)? If yes, the task
manager process does start up and there is another problem. If not,
the task managers seems not to start even.

– Ufuk

On Thu, Jul 7, 2016 at 7:34 AM, Saliya Ekanayake <esaliya@gmail.com> wrote:
> I tried to run more than one task manager per node by duplicating the slave
> IPs. At startup it says for example,
>
> [INFO] 1 instance(s) of taskmanager are already running on j-011.
> Starting taskmanager daemon on host j-011.
>
> but I only see 1 task manager process running.
>
> Is there anything else I need to do?
>
> On Sun, Jul 3, 2016 at 11:28 AM, Ufuk Celebi <uce@apache.org> wrote:
>>
>> Yes, exactly.
>>
>> On Sat, Jul 2, 2016 at 6:28 PM, Saliya Ekanayake <esaliya@gmail.com>
>> wrote:
>> > Thank you, yes, it can be done externally, if not supported within
>> > Flink.
>> >
>> > So the way to spawn multiple task managers would be to list the same
>> > slave
>> > machines N times as necessary in the slaves file?
>> >
>> > On Sat, Jul 2, 2016 at 11:22 AM, Ufuk Celebi <uce@apache.org> wrote:
>> >>
>> >> No, not inside of Flink. That sounds like something like the OS or
>> >> resource manager should handle.
>> >>
>> >> On Sat, Jul 2, 2016 at 5:12 PM, Saliya Ekanayake <esaliya@gmail.com>
>> >> wrote:
>> >> > That's great, so is there support to pin task managers to sockets as
>> >> > well?
>> >> >
>> >> > On Sat, Jul 2, 2016 at 11:08 AM, Ufuk Celebi <uce@apache.org>
wrote:
>> >> >>
>> >> >> Regarding 2) if you don't manually configure something else, that
>> >> >> should happen always.
>> >> >>
>> >> >> Yes, you can run more than one task manager per node depending
on
>> >> >> the
>> >> >> process isolation you want. Within a task manager, there are
>> >> >> multiple
>> >> >> threads for each slot. For example, if you have 2 task managers
with
>> >> >> 2
>> >> >> slots each and submit a job with parallelism 4, each task manager
>> >> >> will
>> >> >> execute 2 sub tasks in separate Threads.
>> >> >>
>> >> >>
>> >> >> On Sat, Jul 2, 2016 at 3:26 AM, Saliya Ekanayake <esaliya@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Ufuk,
>> >> >> >
>> >> >> > Looking at the document you sent it seems only 1 task manager
per
>> >> >> > node
>> >> >> > exist
>> >> >> > and within that you have multiple slots. Is it possible to
run
>> >> >> > more
>> >> >> > than
>> >> >> > 1
>> >> >> > task manager per node? Also, within a task manager is the
>> >> >> > parallelism
>> >> >> > done
>> >> >> > through threads or processes?
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Saliya
>> >> >> >
>> >> >> > On Thu, Jun 30, 2016 at 5:27 PM, Saliya Ekanayake
>> >> >> > <esaliya@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Thank you, I'll check these.
>> >> >> >>
>> >> >> >> In 2.) you said they are likely to exchange through memory.
Is
>> >> >> >> there
>> >> >> >> a
>> >> >> >> case why they wouldn't?
>> >> >> >>
>> >> >> >> On Thu, Jun 30, 2016 at 5:03 AM, Ufuk Celebi <uce@apache.org>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> On Thu, Jun 30, 2016 at 1:44 AM, Saliya Ekanayake
>> >> >> >>> <esaliya@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>> > 1. What parameters are available to control parallelism
within
>> >> >> >>> > a
>> >> >> >>> > node?
>> >> >> >>>
>> >> >> >>> Task Manager processing slots:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-taskmanager-processing-slots
>> >> >> >>>
>> >> >> >>> > 2. Does Flink support shared memory-based messaging
within a
>> >> >> >>> > node
>> >> >> >>> > (without
>> >> >> >>> > doing TCP calls)?
>> >> >> >>>
>> >> >> >>> Yes, local exchanges happen via memory and not TCP,
for example
>> >> >> >>> if
>> >> >> >>> you
>> >> >> >>> have a map-reduce, map subtask 1 and reduce subtask
1 are likely
>> >> >> >>> to
>> >> >> >>> exchange data locally.
>> >> >> >>>
>> >> >> >>> > 3. Is there support for Infiniband interconnect?
>> >> >> >>>
>> >> >> >>> No, not that I'm aware of.
>> >> >> >>>
>> >> >> >>> – Ufuk
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Saliya Ekanayake
>> >> >> >> Ph.D. Candidate | Research Assistant
>> >> >> >> School of Informatics and Computing | Digital Science
Center
>> >> >> >> Indiana University, Bloomington
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Saliya Ekanayake
>> >> >> > Ph.D. Candidate | Research Assistant
>> >> >> > School of Informatics and Computing | Digital Science Center
>> >> >> > Indiana University, Bloomington
>> >> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Saliya Ekanayake
>> >> > Ph.D. Candidate | Research Assistant
>> >> > School of Informatics and Computing | Digital Science Center
>> >> > Indiana University, Bloomington
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Saliya Ekanayake
>> > Ph.D. Candidate | Research Assistant
>> > School of Informatics and Computing | Digital Science Center
>> > Indiana University, Bloomington
>> >
>
>
>
>
> --
> Saliya Ekanayake
> Ph.D. Candidate | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
>

Mime
View raw message