flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eron Wright <eronwri...@gmail.com>
Subject Re: Flink Application Jar file on Docker container
Date Wed, 27 Sep 2017 17:05:57 GMT
There was also a talk about containerization at Flink Forward that touches
on some of your questions.
https://www.youtube.com/watch?v=w721NI-mtAA&t=2s&list=PLDX4T_cnKjD0JeULl1X6iTn7VIkDeYX_X&index=33

Eron

On Wed, Sep 27, 2017 at 9:12 AM, Stefan Richter <s.richter@data-artisans.com
> wrote:

> Hi,
>
> from the top of my head, I cannot see why this should not be possible,
> task managers just need to be able to connect to their job manager.
> Unfortunately, I cannot give a real guarantee here because I am not that
> deeply involved in this aspect of Flink.
>
> Best,
> Stefan
>
> Am 27.09.2017 um 12:39 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:
>
> Hi Stefan,
>
> I have a question in my mid out of curiosity.... Is it possible to run
> flink application within docker container by using flink cluster set up on
> host?
>
> Rahul Raj
>
> On 26 September 2017 at 17:29, Stefan Richter <s.richter@data-artisans.com
> > wrote:
>
>> Hi,
>>
>> if I correctly understood the approach outlined on github, you can start
>> a standalone job manager and the task manager get the JM information either
>> through the provided configuration or through Zookeeper. Take a look at the
>> „running section“, e.g.:
>>
>> 1) „Via Mesos/Marathon: Start a standalone JobManager (you need to
>> replace the flink_recovery_zookeeper_quorum variable with a valid
>> setting for your cluster) [...]“
>> 2) „Via standalone Docker: Start a standalone JobManager (with host
>> networking, binding on 127.0.0.1) […]“
>>
>> Best,
>> Stefan
>>
>>
>> Am 26.09.2017 um 12:43 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:
>>
>> Hi Stefan,
>>
>> Thanks a lot for your answer and sharing the link
>> https://github.com/mesoshq/flink. I went through this and saw its
>> spawning Jobmanager and taskmanager. Now I think, this should be happening.
>> First JobManager will be started on flink cluster on one node, then task
>> manager will be started on another node and both should be running in
>> docker containers on different nodes. Now, my question is how flink's
>> JobManager will get to know about the taskManagers as they are in in
>> different docker containers on different nodes? Will it happen via Mesos?
>>
>> Can we use mesos-appmaster.sh instead which is already built in flink for
>> deployment on mesos?
>>
>> Rahul Raj
>>
>> On 26 September 2017 at 15:32, Stefan Richter <
>> s.richter@data-artisans.com> wrote:
>>
>>> Hi,
>>>
>>> as in my answer to your previous mail, I suggest to take a look at
>>> https://github.com/mesoshq/flink . Unfortunately, there is not yet a
>>> lot documentation about the internals of how this works, so I am also
>>> looping in Till who might know more about specific questions about how
>>> things work together exactly.
>>>
>>> Best,
>>> Stefan
>>>
>>>
>>> Am 26.09.2017 um 09:21 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:
>>>
>>> Currently I have a Flink Application Jar file running on Mesos cluster.
>>> The flink application simply reads data from Kafka and put it to HDFS.
>>>
>>> Now we are planning to create a docker image to  run this application
>>> jar file inside docker containers on Mesos cluster via Marathon.
>>>
>>> Below are the questions that I am looking answers for:
>>>
>>> 1. While building the docker image, how do I include flink-1.3.2 set up
>>> and  my mesos config in flink?
>>>
>>> 2. How shall I run my existing flink application jar?
>>>
>>> 3. Will running my flink application jar on docker containers will run
>>> it on mesos slaves on different docker containers? How docker , Flink,
>>> mesos , Marathon will work together in my case?
>>>
>>> Rahul Raj
>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message