Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B3E07200D0B for ; Wed, 27 Sep 2017 19:06:02 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B263F1609EB; Wed, 27 Sep 2017 17:06:02 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AB3F41609C1 for ; Wed, 27 Sep 2017 19:06:01 +0200 (CEST) Received: (qmail 78802 invoked by uid 500); 27 Sep 2017 17:06:00 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 78790 invoked by uid 99); 27 Sep 2017 17:06:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Sep 2017 17:06:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id ED25BDE1C9 for ; Wed, 27 Sep 2017 17:05:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id B-IUeJ1OS_U8 for ; Wed, 27 Sep 2017 17:05:58 +0000 (UTC) Received: from mail-oi0-f53.google.com (mail-oi0-f53.google.com [209.85.218.53]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 67EDB5FD33 for ; Wed, 27 Sep 2017 17:05:58 +0000 (UTC) Received: by mail-oi0-f53.google.com with SMTP id l74so17725698oih.1 for ; Wed, 27 Sep 2017 10:05:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=e2jttgKk/tKC7wUUx/NCQ0tLU+evi/dYcUahKVWECJw=; b=ZYvGXxNIRh+PkmhFozV9KGufzQ4f3GGH8gX7QsSq1kAVq6/cqWVz/B6l97+4hLVRUZ x98TyszHd1iR9pIeG8iy16iDCwNytwzKGdQTNfooh808idAnDRLRyQDgI54ITPm+q2jI wFd/eyiFTIuFM9Fb0ZvtMfygqEZuDS1ZWmwiSNpRnomAZd0hsGyXtxus0PHsDXaixAQD qL6mpjFnkUMWQrsv4nnPPwgwmqi95gSMMvgUF2s7Xfz/+oLVKijxVH9YutTQ3wiwXPHn zFIkg/eRD1bS22pBBN/qEMdZpqbhacTXEueaufbu3Yp1gLNV+TgQ6bcJrBTJ1IhRKaP6 dQtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=e2jttgKk/tKC7wUUx/NCQ0tLU+evi/dYcUahKVWECJw=; b=sVjVxPOsfe5PHvlSVqOPUAfcXKgjKF9vK032VIi8d2yA+gyPhHzJw5Opf2lalSFDSd VVF1VrSVz5sH1ahGAOzh8lWwfuwYOSOzjdUBaRjmrPRWBPd5y+C1oVGgQzPn7CidtG3T gNY8lVs3p5pGrxbccw+ueTNSrz8IQ3rRCOBIGqmJvyz1MCwf3e6RFe9OanTyxtDJN7R1 xaAZbZDOWboIFAFB4Q/+EdrX7B1SBJlYn2sLM2e1Q8sV8E6k2wTL1tumO9aeq9Emn4cP dN/FkHltpbR/1f2rvJX2j/ADsfkc8Mad8gpUrrsTINjSSnXLFEaltZgk1pzqVhiFvu65 Cr6Q== X-Gm-Message-State: AMCzsaWzHo1YyyAGrY1x/G2R6GxXUBT3GemIdxlf8tOhsHkf0HDG47UA kGErvZy6rU8eZH4VVEQuztwhPz8iuFxbTYTEAv4= X-Google-Smtp-Source: AOwi7QAP4FxCzMTg+4bTOnJ8TPz3jLSdwZPXRptEnMx5B6xWPmn+SUAPwz1wIanYaRtNll8x3xw08He9Jb81FE6V4UU= X-Received: by 10.157.7.166 with SMTP id 35mr1288918oto.490.1506531957608; Wed, 27 Sep 2017 10:05:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.156.87 with HTTP; Wed, 27 Sep 2017 10:05:57 -0700 (PDT) In-Reply-To: References: <33DAE5F6-50CE-48C4-A1E0-97FA8FD9688E@data-artisans.com> From: Eron Wright Date: Wed, 27 Sep 2017 10:05:57 -0700 Message-ID: Subject: Re: Flink Application Jar file on Docker container To: Stefan Richter Cc: Rahul Raj , user , Till Rohrmann Content-Type: multipart/alternative; boundary="94eb2c11ec942d139e055a2ecef7" archived-at: Wed, 27 Sep 2017 17:06:02 -0000 --94eb2c11ec942d139e055a2ecef7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable There was also a talk about containerization at Flink Forward that touches on some of your questions. https://www.youtube.com/watch?v=3Dw721NI-mtAA&t=3D2s&list=3DPLDX4T_cnKjD0Je= ULl1X6iTn7VIkDeYX_X&index=3D33 Eron On Wed, Sep 27, 2017 at 9:12 AM, Stefan Richter wrote: > Hi, > > from the top of my head, I cannot see why this should not be possible, > task managers just need to be able to connect to their job manager. > Unfortunately, I cannot give a real guarantee here because I am not that > deeply involved in this aspect of Flink. > > Best, > Stefan > > Am 27.09.2017 um 12:39 schrieb Rahul Raj : > > Hi Stefan, > > I have a question in my mid out of curiosity.... Is it possible to run > flink application within docker container by using flink cluster set up o= n > host? > > Rahul Raj > > On 26 September 2017 at 17:29, Stefan Richter > wrote: > >> Hi, >> >> if I correctly understood the approach outlined on github, you can start >> a standalone job manager and the task manager get the JM information eit= her >> through the provided configuration or through Zookeeper. Take a look at = the >> =E2=80=9Erunning section=E2=80=9C, e.g.: >> >> 1) =E2=80=9EVia Mesos/Marathon: Start a standalone JobManager (you need = to >> replace the flink_recovery_zookeeper_quorum variable with a valid >> setting for your cluster) [...]=E2=80=9C >> 2) =E2=80=9EVia standalone Docker: Start a standalone JobManager (with h= ost >> networking, binding on 127.0.0.1) [=E2=80=A6]=E2=80=9C >> >> Best, >> Stefan >> >> >> Am 26.09.2017 um 12:43 schrieb Rahul Raj : >> >> Hi Stefan, >> >> Thanks a lot for your answer and sharing the link >> https://github.com/mesoshq/flink. I went through this and saw its >> spawning Jobmanager and taskmanager. Now I think, this should be happeni= ng. >> First JobManager will be started on flink cluster on one node, then task >> manager will be started on another node and both should be running in >> docker containers on different nodes. Now, my question is how flink's >> JobManager will get to know about the taskManagers as they are in in >> different docker containers on different nodes? Will it happen via Mesos= ? >> >> Can we use mesos-appmaster.sh instead which is already built in flink fo= r >> deployment on mesos? >> >> Rahul Raj >> >> On 26 September 2017 at 15:32, Stefan Richter < >> s.richter@data-artisans.com> wrote: >> >>> Hi, >>> >>> as in my answer to your previous mail, I suggest to take a look at >>> https://github.com/mesoshq/flink . Unfortunately, there is not yet a >>> lot documentation about the internals of how this works, so I am also >>> looping in Till who might know more about specific questions about how >>> things work together exactly. >>> >>> Best, >>> Stefan >>> >>> >>> Am 26.09.2017 um 09:21 schrieb Rahul Raj : >>> >>> Currently I have a Flink Application Jar file running on Mesos cluster. >>> The flink application simply reads data from Kafka and put it to HDFS. >>> >>> Now we are planning to create a docker image to run this application >>> jar file inside docker containers on Mesos cluster via Marathon. >>> >>> Below are the questions that I am looking answers for: >>> >>> 1. While building the docker image, how do I include flink-1.3.2 set up >>> and my mesos config in flink? >>> >>> 2. How shall I run my existing flink application jar? >>> >>> 3. Will running my flink application jar on docker containers will run >>> it on mesos slaves on different docker containers? How docker , Flink, >>> mesos , Marathon will work together in my case? >>> >>> Rahul Raj >>> >>> >>> >> >> > > --94eb2c11ec942d139e055a2ecef7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
There was also a talk about containerization at Flink Forw= ard that touches on some of your questions.

Eron
On Wed, Sep 27, 2017 at 9:12 AM, Stefan Richter= <s.richter@data-artisans.com> wrote:
Hi,

from the top of my head, I cannot see why this should not be possible= , task managers just need to be able to connect to their job manager. Unfor= tunately, I cannot give a real guarantee here because I am not that deeply = involved in this aspect of Flink.

Best,
= Stefan

=
Am 27.09.2017 um 12:39 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:
<= br class=3D"m_-8855415385873589818Apple-interchange-newline">
Hi Stefan,

I have a question in my mid out of c= uriosity.... Is it possible to run flink application within docker containe= r by using flink cluster set up on host?

Rahul Raj=

On 26= September 2017 at 17:29, Stefan Richter <s.richter@data-artisan= s.com> wrote:
Hi,

if I correctly understood the= approach outlined on github, you can start a standalone job manager and th= e task manager get the JM information either through the provided configura= tion or through Zookeeper. Take a look at the =E2=80=9Erunning section=E2= =80=9C, e.g.:

1) =E2=80=9EVia Mesos/Marathon: Star= t a standalone JobManager (you need to replace the=C2=A0flink_recovery_zook= eeper_quorum=C2=A0variable with a valid setting for your cluster) [...= ]=E2=80=9C
2) =E2=80=9EVia standalone Docker: Start a standalone = JobManager (with host networking, binding on 127.0.0.1) [=E2=80=A6]=E2=80= =9C

Best,
Stefan


Am 26.09.2017 um 12:43 schrieb Rahul Raj <rahulrajmsrit@gmail.com>:=

Hi Stefan,

Thanks a = lot for your answer and sharing the link https://github.com/mesoshq/flink. I w= ent through this and saw its spawning Jobmanager and taskmanager. Now I thi= nk, this should be happening. First JobManager will be started on flink clu= ster on one node, then task manager will be started on another node and bot= h should be running in docker containers on different nodes. Now, my questi= on is how flink's JobManager will get to know about the taskManagers as= they are in in different docker containers on different nodes? Will it hap= pen via Mesos?

Can we use mesos-appmaster.sh inste= ad which is already built in flink for deployment on mesos?

<= /div>
Rahul Raj =C2=A0

On 26 September 2017 at 15:32, Stefan Richter <s.richter@data-artisans.com> wrote:
Hi,

as= in my answer to your previous mail, I suggest to take a look at=C2=A0https://github.co= m/mesoshq/flink=C2=A0. Unfortunately, there is not yet a lot docum= entation about the internals of how this works, so I am also looping in Til= l who might know more about specific questions about how things work togeth= er exactly.

Best,
Stefan
=C2=A0
Am 26.09.2017 um 09:21 schrieb Rahul Raj &l= t;rahulrajmsri= t@gmail.com>:

Currently I have a Flink Application Jar file running on Mesos cluster.= The flink application simply reads data from Kafka and put it to HDFS.
Now we are planning to create a d= ocker image to=C2=A0 run this application jar file inside docker containers= on Mesos cluster via Marathon.=C2=A0

Below are the questions that I am looking answers for:
<= div dir=3D"auto">
1. While building the docker i= mage, how do I include flink-1.3.2 set up and=C2=A0 my mesos config in flin= k?

2. How shall I run my= existing flink application jar?

3. Will running my flink application jar on docker containers wil= l run it on mesos slaves on different docker containers? How docker , Flink= , mesos , Marathon will work together in my case?
Rahul Raj



--94eb2c11ec942d139e055a2ecef7--