Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1F1ED200C41 for ; Fri, 24 Mar 2017 11:29:53 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 1DD3C160B9A; Fri, 24 Mar 2017 10:29:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1C561160B82 for ; Fri, 24 Mar 2017 11:29:51 +0100 (CET) Received: (qmail 72394 invoked by uid 500); 24 Mar 2017 10:29:51 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 72384 invoked by uid 99); 24 Mar 2017 10:29:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Mar 2017 10:29:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B605718612B for ; Fri, 24 Mar 2017 10:29:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.204 X-Spam-Level: * X-Spam-Status: No, score=1.204 tagged_above=-999 required=6.31 tests=[FREEMAIL_REPLY=1, HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, RP_MATCHES_RCVD=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id sqjA9zheueJ3 for ; Fri, 24 Mar 2017 10:29:48 +0000 (UTC) Received: from smtp.smtpout.orange.fr (smtp05.smtpout.orange.fr [80.12.242.127]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 163B45FB30 for ; Fri, 24 Mar 2017 10:29:47 +0000 (UTC) Received: from [192.168.1.12] ([83.115.121.253]) by mwinf5d61 with ME id zmVl1u00t5U7wPn03mVm2b; Fri, 24 Mar 2017 11:29:47 +0100 X-ME-Helo: [192.168.1.12] X-ME-Auth: cGhpbGlwcGVjYXBhcnJveUBvcmFuZ2UuZnI= X-ME-Date: Fri, 24 Mar 2017 11:29:47 +0100 X-ME-IP: 83.115.121.253 From: Philippe Caparroy Content-Type: multipart/alternative; boundary="Apple-Mail=_D13AD09F-C451-4A7C-B205-42A1B783B144" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: deploying flink cluster in AWS - Containerized Date: Fri, 24 Mar 2017 11:34:15 +0100 References: <83D9B129-7D14-447B-ADF6-4120C7EDDB3E@orange.fr> To: user@flink.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3124) archived-at: Fri, 24 Mar 2017 10:29:53 -0000 --Apple-Mail=_D13AD09F-C451-4A7C-B205-42A1B783B144 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Weave allows encryption of the vpn, and your Flink containers can be = secured using kerberos = https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/config.h= tml#kerberos-based-security. > Le 24 mars 2017 =C3=A0 11:16, Chakravarthy varaga = a =C3=A9crit : >=20 > Hi, >=20 > Thanks for your inputs. It kind of makes sense to use a container = orchestrator to plough through networking under the hood. > How do you tackle security? >=20 > I don't see a way to authorize users for job management. I = understand few orchestrators provide name space isolation and security = policies on these. How do this work if the flink cluster is standalone = on AWS ? > =20 >=20 > Best Regards > CVP >=20 > On Fri, Mar 24, 2017 at 8:49 AM, Philippe Caparroy = > = wrote: > Hi, >=20 > If I can give my 2 cents. >=20 > One simple solution to your problem is using weave = (https://www.weave.works/ ) a Docker network = plugin. >=20 > We=E2=80=99ve been working for more then year with dockerized = (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using = weave. >=20 > Design your docker container so that you can set the cluster size on = startup (number of task manager stand job managers should be a docker = arg). >=20 > Weave will act as a switch with dns server embedded. Your containers = will only have to be configured with hosts names such as : = flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local, = flink.jobmanager-1.weave.local, and so on =E2=80=A6 >=20 > with flink Yarn it=E2=80=99s even simpler, but you have to dockerize a = Yarn cluster. >=20 > It works perfectly on bare metal machines and in the cloud = (digital-ocean, aws,=E2=80=A6). >=20 >=20 >=20 >> Le 24 mars 2017 =C3=A0 08:50, Chakravarthy varaga = > a =C3=A9crit = : >>=20 >> Hi, >>=20 >> I request someone to help here. >>=20 >> Best Regards >> CVP >>=20 >> On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga = > wrote: >> I'm looking forward to hearing some updates on this... >>=20 >> Any help here is highly appreciated !! >>=20 >> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga = > wrote: >> Hi Team, >>=20 >> We are doing a PoC to deploy Flink cluster on AWS. All runtime = components will be dockerized. >> =20 >> I have few questions in relation to discover & security: >>=20 >> 1. How does Job Manager discover task managers? Do they talk to = over TCP ? >>=20 >> 2. If the runtime components TM, JM are containerized how are = the IPs resolved dynamically? Basically do I have to configure the JM = with the hostnames of the TMs. If so, if the TMs are on ephemeral IPs = and on restart of TM how does the job manager know the TM's (IP/Host). = Before I go into DNS and subnets, I'd like to understand how they = disvoer & talk to each other ! >>=20 >> 3. I went through some Flink materials on the web on security = precisely on kerebros. However how do I ensure that user level = authentication is applied on job management. For ex., only certain users = are allowed to start/stop jobs ? This question is in relation to if = flink is deployed as standalone-cluster >> =20 >> Thanks & Regards >> CVP >>=20 >>=20 >=20 >=20 --Apple-Mail=_D13AD09F-C451-4A7C-B205-42A1B783B144 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Weave  allows encryption of the vpn, and your Flink = containers can be secured using kerberos https://ci.apache.org/projects/flink/flink-docs-release-1.2/set= up/config.html#kerberos-based-security.

Le = 24 mars 2017 =C3=A0 11:16, Chakravarthy varaga <chakravarthyvp@gmail.com> a =C3=A9crit :

Hi,

     Thanks for = your inputs. It kind of makes sense to use a container orchestrator to = plough through networking under the hood.
     How do you tackle security?

     = I don't see a way to authorize users for job management. I understand = few orchestrators provide name space isolation and security policies on = these. How do this work if the flink cluster is standalone on AWS ?
    

Best Regards
CVP

On Fri, Mar 24, 2017 at 8:49 AM, = Philippe Caparroy <philippe.caparroy@orange.fr> wrote:
Hi,

If I can give my 2 cents.

One simple solution to = your problem is using weave (https://www.weave.works/) a Docker = network plugin.

We=E2=80=99ve been working for more then year with dockerized = (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) = cluster using weave.

Design your docker container so that you can set the cluster = size on startup (number of task manager stand job managers should be a = docker arg).

Weave will act as a switch with dns server embedded. Your = containers will only have to be configured with hosts names such as : = flink.taskmanager-1.weave.local, = link.taskmanager-2.weave.local, = flink.jobmanager-1.weave.local, and so on =E2=80=A6
<= div class=3D"">
with flink Yarn = it=E2=80=99s even simpler, but you have to dockerize a Yarn = cluster.

It = works perfectly on bare metal machines and in the cloud (digital-ocean, = aws,=E2=80=A6).

Le 24 mars 2017 =C3=A0 08:50, Chakravarthy varaga <chakravarthyvp@gmail.com> a =C3=A9crit :

Hi,

    I request someone to help here.

Best Regards
CVP

On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy = varaga <chakravarthyvp@gmail.com> wrote:
I'm looking forward to hearing some updates = on this...

Any help here is highly = appreciated !!

On Thu, Mar 23, 2017 at 4:20 PM, = Chakravarthy varaga <chakravarthyvp@gmail.com> wrote:
Hi Team,

     We are doing a = PoC to deploy Flink cluster on AWS. All runtime components will be = dockerized.
     =
      I = have few questions in relation to discover & security:

      1. How does Job Manager = discover task managers? Do they talk to over TCP ?

      2. If = the runtime components TM, JM are containerized how are the IPs resolved = dynamically? Basically do I have to configure the JM with the hostnames = of the TMs. If so, if the TMs are on ephemeral IPs and on restart of TM = how does the job manager know the TM's (IP/Host). Before I go into DNS = and subnets, I'd like to understand how they disvoer & talk to each = other !

       3. I went through some = Flink materials on the web on security precisely on kerebros. However = how do I ensure that user level authentication is applied on job = management. For ex., only certain users are allowed to start/stop jobs ? = This question is in relation to if flink is deployed as = standalone-cluster
  
Thanks & Regards
CVP





= --Apple-Mail=_D13AD09F-C451-4A7C-B205-42A1B783B144--