Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <33DAE5F6-50CE-48C4-A1E0-97FA8FD9688E@data-artisans.com>
References: <CAKcRf45hxTNd8iqzBk6fickr0YOQPTQy8ug4DnKVf0y_PCBcdg@mail.gmail.com>
 <CAKcRf47HnfTsZzsUD7Ljzs2Lcmb6JusPuW8Gw=zsVLO3uM15Gg@mail.gmail.com>
 <CAKcRf44nU7Rx5eYA4pM8H4+Q+mw6pWmDXf0b8x5V8Og9WZzhdQ@mail.gmail.com>
 <CAKcRf45+jC4wzJN=19z3D0KVt9busamNgmyJ1jmeVJrbbifz9w@mail.gmail.com>
 <CAKcRf47=gS6ug77oFLnXvSNEnsKT1BHc4pse6HXV2NZYPY7oyw@mail.gmail.com>
 <CAKcRf447tyUEOSZTwFmO2KA-z_=FSfHsJL7bkQBrdFs_dfvYsg@mail.gmail.com>
 <CAKcRf45M9kwaV3=xEdDiwdomJJDkewPOC8Vx74sPG8jcaRR8bw@mail.gmail.com>
 <CAKcRf45c9bsZ-9JA4vhT2dreEvxhcW69OpqSWywpxTQq=3DcuQ@mail.gmail.com>
 <CAKcRf45mCDEF6vN_qKsznSU969a60hjWzyX0RSfSPre1SXczDw@mail.gmail.com>
 <CAKcRf45wepuwu4p8WdKCht0K6hrtbn+gS4HFqbjeWusVxQD-YQ@mail.gmail.com>
 <CAKcRf45s4PPXTz+ztzU8KaymGhoShgtaOagOZHERQPq_wrA56g@mail.gmail.com>
 <CAKcRf47Mv8pWkE=y+oPEfN9oJ=DC2j369mngg-i+w4-NEnfxPQ@mail.gmail.com>
 <CAKcRf44tZp4cOykgYcuUH5s3FyGVGyAQ1+vNvkc-D1birRtreA@mail.gmail.com>
 <CAKcRf47SB1yN_==bJn+V05Skk=qr0NFTRHPGo=o0KNcTjN797g@mail.gmail.com>
 <CAKcRf46ymVvzo77c+1bArz9+qSmy8cxYP7OCD5Scw6cknjMmaA@mail.gmail.com>
 <CAKcRf441dHoCmNpPJqDH8brNtee+UYQuM+f0j-YyzxPgR5Xb-w@mail.gmail.com>
 <CAKcRf45WGbpDsUmizBr2h0L_+MHCpsWZ6nWV_0_3H-xSMrHVaw@mail.gmail.com>
 <CAKcRf44NiZGmnyOfxj1+h567r8zCA1U1O-SEf24s-RnPFKu3AA@mail.gmail.com>
 <CAKcRf46n78dENQn1HXMGUCJppC_-NioozUBsA9CVx67=u-3OUg@mail.gmail.com>
 <CAKcRf45Yzb1vdWqoOYaz_-UqSOa5oMzCKLTQTejjW=70iSFyhQ@mail.gmail.com>
 <CAKcRf443BhVAiDp5jusD5wxyTsD-L6T6eQczsKYubjChwYdgZA@mail.gmail.com> <33DAE5F6-50CE-48C4-A1E0-97FA8FD9688E@data-artisans.com>
From: Rahul Raj <rahulrajmsrit@gmail.com>
Date: Tue, 26 Sep 2017 16:13:38 +0530
Message-ID: <CAKcRf47gDFSPiBU-L-+gr0dQMNS3gYsQbWB2FR9HOUPve7bmgA@mail.gmail.com>
Subject: Re: Flink Application Jar file on Docker container
To: Stefan Richter <s.richter@data-artisans.com>
Cc: user <user@flink.apache.org>, Till Rohrmann <till@data-artisans.com>
Content-Type: multipart/alternative; boundary="001a1144302815bd62055a155929"
archived-at: Tue, 26 Sep 2017 10:43:44 -0000

--001a1144302815bd62055a155929
Content-Type: text/plain; charset="UTF-8"

Hi Stefan,

Thanks a lot for your answer and sharing the link
https://github.com/mesoshq/flink. I went through this and saw its spawning
Jobmanager and taskmanager. Now I think, this should be happening. First
JobManager will be started on flink cluster on one node, then task manager
will be started on another node and both should be running in docker
containers on different nodes. Now, my question is how flink's JobManager
will get to know about the taskManagers as they are in in different docker
containers on different nodes? Will it happen via Mesos?

Can we use mesos-appmaster.sh instead which is already built in flink for
deployment on mesos?

Rahul Raj

On 26 September 2017 at 15:32, Stefan Richter <s.richter@data-artisans.com>
wrote:

> Hi,
>
> as in my answer to your previous mail, I suggest to take a look at
> https://github.com/mesoshq/flink . Unfortunately, there is not yet a lot
> documentation about the internals of how this works, so I am also looping
> in Till who might know more about specific questions about how things work
> together exactly.
>
> Best,
> Stefan
>
>
> Am 26.09.2017 um 09:21 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:
>
> Currently I have a Flink Application Jar file running on Mesos cluster.
> The flink application simply reads data from Kafka and put it to HDFS.
>
> Now we are planning to create a docker image to  run this application jar
> file inside docker containers on Mesos cluster via Marathon.
>
> Below are the questions that I am looking answers for:
>
> 1. While building the docker image, how do I include flink-1.3.2 set up
> and  my mesos config in flink?
>
> 2. How shall I run my existing flink application jar?
>
> 3. Will running my flink application jar on docker containers will run it
> on mesos slaves on different docker containers? How docker , Flink, mesos ,
> Marathon will work together in my case?
>
> Rahul Raj
>
>
>

--001a1144302815bd62055a155929
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Stefan,<div><br></div><div>Thanks a lot for your answer=
 and sharing the link <a href=3D"https://github.com/mesoshq/flink">https://=
github.com/mesoshq/flink</a>. I went through this and saw its spawning Jobm=
anager and taskmanager. Now I think, this should be happening. First JobMan=
ager will be started on flink cluster on one node, then task manager will b=
e started on another node and both should be running in docker containers o=
n different nodes. Now, my question is how flink&#39;s JobManager will get =
to know about the taskManagers as they are in in different docker container=
s on different nodes? Will it happen via Mesos?</div><div><br></div><div>Ca=
n we use mesos-appmaster.sh instead which is already built in flink for dep=
loyment on mesos?</div><div><br></div><div>Rahul Raj =C2=A0</div></div><div=
 class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 26 September 2017 =
at 15:32, Stefan Richter <span dir=3D"ltr">&lt;<a href=3D"mailto:s.richter@=
data-artisans.com" target=3D"_blank">s.richter@data-artisans.com</a>&gt;</s=
pan> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break=
-word">Hi,<div><br></div><div>as in my answer to your previous mail, I sugg=
est to take a look at=C2=A0<a href=3D"https://github.com/mesoshq/flink" tar=
get=3D"_blank">https://github.com/mesoshq/<wbr>flink</a>=C2=A0. Unfortunate=
ly, there is not yet a lot documentation about the internals of how this wo=
rks, so I am also looping in Till who might know more about specific questi=
ons about how things work together exactly.</div><div><br></div><div>Best,<=
/div><div>Stefan</div><div><div class=3D"h5"><div>=C2=A0<br><div><blockquot=
e type=3D"cite"><div>Am 26.09.2017 um 09:21 schrieb Rahul Raj &lt;<a href=
=3D"mailto:rahulrajmsrit@gmail.com" target=3D"_blank">rahulrajmsrit@gmail.c=
om</a>&gt;:</div><br class=3D"m_-8191869656293146598Apple-interchange-newli=
ne"><div><div dir=3D"auto">Currently I have a Flink Application Jar file ru=
nning on Mesos cluster. The flink application simply reads data from Kafka =
and put it to HDFS.<div dir=3D"auto"><br></div><div dir=3D"auto">Now we are=
 planning to create a docker image to=C2=A0 run this application jar file i=
nside docker containers on Mesos cluster via Marathon.=C2=A0</div><div dir=
=3D"auto"><br></div><div dir=3D"auto">Below are the questions that I am loo=
king answers for:</div><div dir=3D"auto"><br></div><div dir=3D"auto">1. Whi=
le building the docker image, how do I include flink-1.3.2 set up and=C2=A0=
 my mesos config in flink?</div><div dir=3D"auto"><br></div><div dir=3D"aut=
o">2. How shall I run my existing flink application jar?</div><div dir=3D"a=
uto"><br></div><div dir=3D"auto">3. Will running my flink application jar o=
n docker containers will run it on mesos slaves on different docker contain=
ers? How docker , Flink, mesos , Marathon will work together in my case?</d=
iv><div dir=3D"auto"><br></div><div dir=3D"auto">Rahul Raj</div></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br=
></div>

--001a1144302815bd62055a155929--