flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: Flink Application Jar file on Docker container
Date Tue, 26 Sep 2017 11:59:58 GMT
Hi,

if I correctly understood the approach outlined on github, you can start a standalone job
manager and the task manager get the JM information either through the provided configuration
or through Zookeeper. Take a look at the „running section“, e.g.:

1) „Via Mesos/Marathon: Start a standalone JobManager (you need to replace the flink_recovery_zookeeper_quorum
variable with a valid setting for your cluster) [...]“
2) „Via standalone Docker: Start a standalone JobManager (with host networking, binding
on 127.0.0.1) […]“

Best,
Stefan


> Am 26.09.2017 um 12:43 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:
> 
> Hi Stefan,
> 
> Thanks a lot for your answer and sharing the link https://github.com/mesoshq/flink <https://github.com/mesoshq/flink>.
I went through this and saw its spawning Jobmanager and taskmanager. Now I think, this should
be happening. First JobManager will be started on flink cluster on one node, then task manager
will be started on another node and both should be running in docker containers on different
nodes. Now, my question is how flink's JobManager will get to know about the taskManagers
as they are in in different docker containers on different nodes? Will it happen via Mesos?
> 
> Can we use mesos-appmaster.sh instead which is already built in flink for deployment
on mesos?
> 
> Rahul Raj  
> 
> On 26 September 2017 at 15:32, Stefan Richter <s.richter@data-artisans.com <mailto:s.richter@data-artisans.com>>
wrote:
> Hi,
> 
> as in my answer to your previous mail, I suggest to take a look at https://github.com/mesoshq/flink
<https://github.com/mesoshq/flink> . Unfortunately, there is not yet a lot documentation
about the internals of how this works, so I am also looping in Till who might know more about
specific questions about how things work together exactly.
> 
> Best,
> Stefan
>  
>> Am 26.09.2017 um 09:21 schrieb Rahul Raj <rahulrajmsrit@gmail.com <mailto:rahulrajmsrit@gmail.com>>:
>> 
>> Currently I have a Flink Application Jar file running on Mesos cluster. The flink
application simply reads data from Kafka and put it to HDFS.
>> 
>> Now we are planning to create a docker image to  run this application jar file inside
docker containers on Mesos cluster via Marathon. 
>> 
>> Below are the questions that I am looking answers for:
>> 
>> 1. While building the docker image, how do I include flink-1.3.2 set up and  my mesos
config in flink?
>> 
>> 2. How shall I run my existing flink application jar?
>> 
>> 3. Will running my flink application jar on docker containers will run it on mesos
slaves on different docker containers? How docker , Flink, mesos , Marathon will work together
in my case?
>> 
>> Rahul Raj
> 
> 


Mime
View raw message