Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E757F18668 for ; Tue, 12 Jan 2016 10:56:20 +0000 (UTC) Received: (qmail 44091 invoked by uid 500); 12 Jan 2016 10:56:20 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 44004 invoked by uid 500); 12 Jan 2016 10:56:20 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 43984 invoked by uid 99); 12 Jan 2016 10:56:20 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2016 10:56:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 4385EC1833 for ; Tue, 12 Jan 2016 10:56:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id MeVrW9W35vhq for ; Tue, 12 Jan 2016 10:56:10 +0000 (UTC) Received: from mail-io0-f170.google.com (mail-io0-f170.google.com [209.85.223.170]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 5798120479 for ; Tue, 12 Jan 2016 10:56:10 +0000 (UTC) Received: by mail-io0-f170.google.com with SMTP id 77so348271202ioc.2 for ; Tue, 12 Jan 2016 02:56:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=yYDJYpvfEob2pRynw+vogXo5y7biRWeYFewc4+0FVeg=; b=fVm+egIqakoLjGiu6U2mDQmD+FbNraeo1jLl65qluEB+TdFhYfGtxVkqluxCHOS90t 35vCnCfYoBYmUgZRJa2PO8sueVQIXVEJdFFLajU7RWPK41v8Y3/zCCflovmY22Td/jrv S4w58QOwqf3+Fx4fcW2QoSf6GivYrZ6pVk6F8YNZs1sDC29V7DPN9dATI3pTJyKzwHG8 zB3M8ae3pUKE2oTLenxF0mtfQuR/jZPz380PLxCtTkiAvT03Ww+FEtEIF3qNZLlZAd0V RAHqgbGHTI5XeiR+hjTjjobiHdchjvqEFOl9x83k3mgiXCZixFb3cISUJZC8KdOm9Q3x Fg4Q== MIME-Version: 1.0 X-Received: by 10.107.63.214 with SMTP id m205mr143993188ioa.116.1452596169760; Tue, 12 Jan 2016 02:56:09 -0800 (PST) Sender: ewenstephan@gmail.com Received: by 10.107.159.194 with HTTP; Tue, 12 Jan 2016 02:56:09 -0800 (PST) In-Reply-To: References: Date: Tue, 12 Jan 2016 11:56:09 +0100 X-Google-Sender-Auth: IE7O8v62pJ_nMLOJRiUGznniV8I Message-ID: Subject: Re: Security in Flink From: Stephan Ewen To: user@flink.apache.org Content-Type: multipart/alternative; boundary=94eb2c062a4cb36510052920e66f --94eb2c062a4cb36510052920e66f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Welly! In the end, all remote communication in Flink will go through Netty (Flink direct shuffles do, and Akka uses Netty as well). Netty authenticates connections and encrypts data via SSL, implementing Java's SSLContext. As far as I know, the available algorithms for encryption / signatures are the usual ones in the TLS standard (can also be configured via JVM arguments). Greetings, Stephan On Tue, Jan 12, 2016 at 9:38 AM, Ufuk Celebi wrote: > Hey Welly! > > I=E2=80=99m not aware of any concrete plans, but is it possible that you = share > your requirements on a high level? > > =E2=80=93 Ufuk > > > On 12 Jan 2016, at 08:24, Welly Tambunan wrote: > > > > Hi Stephen, > > > > Do you have any plan on which encryption method and mechanism will be > used on Flink ? Could you share about the detail on this ? > > > > We have very strict requirement from client that every communication > need to be encryption. So any detail would be really appreciated for > answering their security concern. > > > > > > Cheers > > > > On Mon, Jan 11, 2016 at 9:46 PM, Sourav Mazumder < > sourav.mazumder00@gmail.com> wrote: > > Thanks Steven for your details response. Things are more clear to me no= w. > > > > A follow up Qs - > > Looks like most of the security support depends on Hadoop ? What happen= s > if anyone wants to use Flink with Hadoop (in a cluster where Hadoop is no= t > there) ? > > > > Regards, > > Sourav > > > > On Sun, Jan 10, 2016 at 12:41 PM, Stephan Ewen wrote= : > > Hi Sourav! > > > > There is user-authentication support in Flink via the Hadoop / Kerberos > infrastructure. If you run Flink on YARN, it should seamlessly work that > Flink acquires the Kerberos tokens of the user that submits programs, and > authenticate itself at YARN, HDFS, and HBase with that. > > > > If you run Flink standalone, Flink can still authenticate at HDFS/HBase > via Kerberos, with a bit of manual help by the user (running kinit on the > workers). > > > > With Kafka 0.9 and Flink's upcoming connector ( > https://github.com/apache/flink/pull/1489), streaming programs can > authenticate themselves as stream brokers via SSL (and read via encrypted > connections). > > > > > > What we have on the roadmap for the coming months it the following: > > - Encrypt in-flight data streams that are exchanged between worker > nodes (TaskManagers). > > - Encrypt the coordination messages between client/master/workers. > > Note that these refer to encryption between Flink's own components only= , > which would use transient keys generated just for a specific job or sessi= on > (hence would not need any user involvement). > > > > > > Let us know if that answers your questions, and if that meets your > requirements. > > > > Greetings, > > Stephan > > > > > > On Fri, Jan 8, 2016 at 3:23 PM, Sourav Mazumder < > sourav.mazumder00@gmail.com> wrote: > > Hi, > > > > Can anyone point me to ant documentation on support for Security in > Flink ? > > > > The type of information I'm looking for are - > > > > 1. How do I do user level authentication to ensure that a job is > submitted/deleted/modified by the right user ? Is it possible though the > web client ? > > 2. Authentication across multiple slave nodes (where the task managers > are running) and driver program so that they can communicate with each ot= her > > 3. Support for SSL/encryption for data exchanged happening across the > slave nodes > > 4. Support for pluggable authentication with existing solution like LDA= P > > > > If not there today is there a roadmap for these security features ? > > > > Regards, > > Sourav > > > > > > > > > > > > -- > > Welly Tambunan > > Triplelands > > > > http://weltam.wordpress.com > > http://www.triplelands.com > > --94eb2c062a4cb36510052920e66f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Welly!

In the end, all remote commun= ication in Flink will go through Netty (Flink direct shuffles do, and Akka = uses Netty as well).

Netty authenticates connectio= ns and encrypts data via SSL, implementing Java's SSLContext.

As far as I know, the available algorithms for encryption /= signatures are the usual ones in the TLS standard (can also be configured = via JVM arguments).

Greetings,
Stephan


On Tue, Jan 12, 2016 at 9:38 AM, Ufuk Celebi <= uce@apache.org><= /span> wrote:
Hey Welly!

I=E2=80=99m not aware of any concrete plans, but is it possible that you sh= are your requirements on a high level?

=E2=80=93 Ufuk

> On 12 Jan 2016, at 08:24, Welly Tambunan <if05041@gmail.com> wrote:
>
> Hi Stephen,
>
> Do you have any plan on which encryption method and mechanism will be = used on Flink ? Could you share about the detail on this ?
>
> We have very strict requirement from client that every communication n= eed to be encryption. So any detail would be really appreciated for answeri= ng their security concern.
>
>
> Cheers
>
> On Mon, Jan 11, 2016 at 9:46 PM, Sourav Mazumder <sourav.mazumder00@gmail.com> wrote: > Thanks Steven for your details response. Things are more clear to me n= ow.
>
> A follow up Qs -
> Looks like most of the security support depends on Hadoop ? What happe= ns if anyone wants to use Flink with Hadoop (in a cluster where Hadoop is n= ot there) ?
>
> Regards,
> Sourav
>
> On Sun, Jan 10, 2016 at 12:41 PM, Stephan Ewen <sewen@apache.org> wrote:
> Hi Sourav!
>
> There is user-authentication support in Flink via the Hadoop / Kerbero= s infrastructure. If you run Flink on YARN, it should seamlessly work that = Flink acquires the Kerberos tokens of the user that submits programs, and a= uthenticate itself at YARN, HDFS, and HBase with that.
>
> If you run Flink standalone, Flink can still authenticate at HDFS/HBas= e via Kerberos, with a bit of manual help by the user (running kinit on the= workers).
>
> With Kafka 0.9 and Flink's upcoming connector (htt= ps://github.com/apache/flink/pull/1489), streaming programs can authent= icate themselves as stream brokers via SSL (and read via encrypted connecti= ons).
>
>
> What we have on the roadmap for the coming months it the following: >=C2=A0 =C2=A0- Encrypt in-flight data streams that are exchanged betwee= n worker nodes (TaskManagers).
>=C2=A0 =C2=A0- Encrypt the coordination messages between client/master/= workers.
> Note that these refer to encryption between Flink's own components= only, which would use transient keys generated just for a specific job or = session (hence would not need any user involvement).
>
>
> Let us know if that answers your questions, and if that meets your req= uirements.
>
> Greetings,
> Stephan
>
>
> On Fri, Jan 8, 2016 at 3:23 PM, Sourav Mazumder <sourav.mazumder00@gmail.com> wrote:
> Hi,
>
> Can anyone point me to ant documentation on support for Security in Fl= ink ?
>
> The type of information I'm looking for are -
>
> 1. How do I do user level authentication to ensure that a job is submi= tted/deleted/modified by the right user ? Is it possible though the web cli= ent ?
> 2. Authentication across multiple slave nodes (where the task managers= are running) and driver program so that they can communicate with each oth= er
> 3. Support for SSL/encryption for data exchanged happening across the = slave nodes
> 4. Support for pluggable authentication with existing solution like LD= AP
>
> If not there today is there a roadmap for these security features ? >
> Regards,
> Sourav
>
>
>
>
>
> --
> Welly Tambunan
> Triplelands
>
> http://weltam.wordpress.com
> http://www.triplelands.com


--94eb2c062a4cb36510052920e66f--