flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Adding TaskManager's
Date Thu, 18 Feb 2016 07:54:21 GMT
No, the "slaves" file is still used to ssh into the machines and start
the task manager processes in the start-cluster.sh script. So you
still need password-less ssh into the machines if you want to use
that. The task managers discover the job manager via ZooKeeper though
(therefore you don't need to configure the jobmanager address in the
config).

In theory, you can also skip the "slaves" file if you ssh manually
into the machines and start the task managers via the taskmanger.sh
script, but I don't think that this is what you are looking for. Or
are you?

– Ufuk


On Wed, Feb 17, 2016 at 10:27 PM, Deepak Jha <dkjhanitt@gmail.com> wrote:
> Hi Max and Stephan,
> Does this mean that I can start Flink HA cluster without keeping any entry
> in "slaves" file ? I'm asking this because then I should not worry about
> copying public key for password-less ssh in Flink HA cluster....
>
> On Wed, Feb 17, 2016 at 12:38 PM, Deepak Jha <dkjhanitt@gmail.com> wrote:
>
>> Sorry for the typo Stephan
>>
>>
>> On Wednesday, February 17, 2016, Deepak Jha <dkjhanitt@gmail.com> wrote:
>>
>>> Thanks Max and Steven for the response.
>>>
>>> On Wednesday, February 17, 2016, Stephan Ewen <sewen@apache.org> wrote:
>>>
>>>> Hi Deepak!
>>>>
>>>> The "slaves" file is only used by the SSH script to start a standalone
>>>> cluster.
>>>>
>>>> As Max said, TaskManagers register dynamically at the JobManager.
>>>>
>>>> Discovery works via:
>>>>    - config in non-HA mode
>>>>    - ZooKeeper in HA mode
>>>>
>>>>
>>>>
>>>> On Wed, Feb 17, 2016 at 10:11 AM, Maximilian Michels <mxm@apache.org>
>>>> wrote:
>>>>
>>>> > Hi Deepak,
>>>> >
>>>> > The job manager doesn't have to know about task managers. They will
>>>> > simply register at the job manager using the provided configuration.
>>>> > In HA mode, they will lookup the currently leading job manager first
>>>> > and then connect to it. The job manager can then assign work.
>>>> >
>>>> > Cheers,
>>>> > Max
>>>> >
>>>> > On Tue, Feb 16, 2016 at 10:41 PM, Deepak Jha <dkjhanitt@gmail.com>
>>>> wrote:
>>>> > > Hi All,
>>>> > > I have a question on scaling-up/scaling-down flink cluster.
>>>> > > As per the documentation, in order to scale-up the cluster, I can
>>>> add a
>>>> > new
>>>> > > taskmanager on the fly and jobmanager can assign work to it.
>>>> Assuming, I
>>>> > > have Flink HA , so in the event of master JobManager failure, how
is
>>>> this
>>>> > > taskmanager detail is going to get transferred ? I believe new
master
>>>> > will
>>>> > > just read the contents from slaves config file. Can anyone give
more
>>>> > > clarity on how this is done ? Or, Is it union of slaves and the
>>>> > > taskmanager's that are added on the fly ?
>>>> > >
>>>> > > --
>>>> > > Thanks,
>>>> > > Deepak
>>>> >
>>>>
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
>
> --
> Thanks,
> Deepak Jha

Mime
View raw message