Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C6A8D200C80 for ; Thu, 25 May 2017 18:27:40 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C515C160BCA; Thu, 25 May 2017 16:27:40 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 97927160BC7 for ; Thu, 25 May 2017 18:27:39 +0200 (CEST) Received: (qmail 42653 invoked by uid 500); 25 May 2017 16:27:38 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 42643 invoked by uid 99); 25 May 2017 16:27:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 May 2017 16:27:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 36B641AFE3A for ; Thu, 25 May 2017 16:27:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.4 X-Spam-Level: X-Spam-Status: No, score=-0.4 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id hBR87ei53KOU for ; Thu, 25 May 2017 16:27:36 +0000 (UTC) Received: from mail-oi0-f48.google.com (mail-oi0-f48.google.com [209.85.218.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B470360E04 for ; Thu, 25 May 2017 16:27:30 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id w10so287076665oif.0 for ; Thu, 25 May 2017 09:27:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=9gN2TNlEWFU4EevKvu6NBzPySV28zpl5EvoR9JPszJ4=; b=HrDXQ57jc4Dw3aztX2qKBFUtbW8DoVFe/piIGSKnysvgBP9ZYmRE6jqfmsQPpPad21 dyBUpbTPCaxs5BkDL/tDPwnj6uubKdGOJpkjjDZxNG5o6TCLfIDbB6H1625vmHoJiChx P9yQ+HwUlbrAk9dgAR7/bGs7t+MYrvGq9yChEmbJxizaf1nIt87Mls3Cl97fzWxVmJia 9Uy3ScV2ofRfPdXjjDSKJ/XN8ML+bKR+EOyxn9A5wN3mq7N7++lqASgBwI8Mgmol6nIB OleeUf7yG0jaDP8IulAcjxGOMgVcm1o33Ex73SO1pvbOK2UWMJsXyqHpUvdxUlU3Aq69 wpjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=9gN2TNlEWFU4EevKvu6NBzPySV28zpl5EvoR9JPszJ4=; b=n7FYMGAN4uEsx+2U3lD3i/q1KEums8JGCtPOtkMusUaAsdoCeqWvLPEM0o7E9Y2Ejx Ypst5lLSZaPSKjIvBSY0oMHNLY228y0rU3LJR3EIrqjupo6UImwehI3eJvwzOupXGjkr bTEdCaAuN/9yoyfeqYK/Er+bbsBkRN/LXJbJaEcmr4oHz/YYxKXoaqeyXlal8tliVjPj KO94i6Ipou4FTEPcaajlbv1ggtCrnzIth9HqHlC2JPf8FSQXVJ+azeVF/avqQ7aaILbN fUZ32BDBP1QH3ROorgo64CryfL6vd/Qbx9JNEOwPakoTXgOtmFzg5fC0ENSpgmnohOtC zk/Q== X-Gm-Message-State: AODbwcB1y11600OKpin8O9orB+rzukpb7wrgVVGsmS8VyflWNHMzvPNh D8f84G2srM8cV8xllMeflKZcUj06AQ== X-Received: by 10.157.56.140 with SMTP id p12mr8019547otc.206.1495729649503; Thu, 25 May 2017 09:27:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.89.9 with HTTP; Thu, 25 May 2017 09:27:29 -0700 (PDT) In-Reply-To: References: From: Eron Wright Date: Thu, 25 May 2017 09:27:29 -0700 Message-ID: Subject: Re: Kafka 0.10 jaas multiple clients To: "Tzu-Li (Gordon) Tai" Cc: user@flink.apache.org Content-Type: multipart/alternative; boundary="001a11c017b27052df05505bb289" archived-at: Thu, 25 May 2017 16:27:41 -0000 --001a11c017b27052df05505bb289 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Gordon's suggestion seems like a good way to provide per-job credentials based on application-specific properties. In contrast, Flink's built-in JAAS features are aimed at making the Flink cluster's Kerberos credentials available to jobs. I want to reiterate that all jobs (for a given Flink cluster) run with full privilege in the JVM, and that Flink does not guarantee isolation (of security information) between jobs. My suggestion is to not run untrusted job code in a shared Flink cluster. On Wed, May 24, 2017 at 8:46 PM, Tzu-Li (Gordon) Tai wrote: > Hi Gwenhael, > > Follow-up for this: > > Turns out what you require is already available with Kafka 0.10, using > dynamic JAAS configurations [1] instead of a static JAAS file like what > you=E2=80=99re currently doing. > > The main thing to do is to set a =E2=80=9Csasl.jaas.config=E2=80=9D in th= e config > properties for your individual Kafka consumer / producer. > This will override any static JAAS configuration used. > Note 2 things here: 1) static JAAS configurations are a JVM process-wide > installation, meaning using that any separate Kafka client within the sam= e > process can always only share the same credentials and 2) the =E2=80=9CKa= fkaClient=E2=80=9D > is a fixed JAAS lookup section key that the Kafka clients use, which I > don=E2=80=99t think is modifiable. So using the static config approach wo= uld never > work. > > An example =E2=80=9Csasl.jaas.config=E2=80=9D for plain logins: > "org.apache.kafka.common.security.plain.PlainLoginModule required > username=3Dxxxx password=3Dyyyy > > Simply have different values for each of the Kafka consumer / producers > you=E2=80=99re using. > > Cheers, > Gordon > > > On 8 May 2017 at 4:42:07 PM, Tzu-Li (Gordon) Tai (tzulitai@apache.org) > wrote: > > Hi Gwenhael, > > Sorry for the very long delayed response on this. > > As you noticed, the =E2=80=9CKafkaClient=E2=80=9D entry name seems to be = a hardcoded thing > on the Kafka side, so currently I don=E2=80=99t think what you=E2=80=99re= asking for is > possible. > > It seems like this could be made possible with some of the new > authentication features in Kafka 0.10 that seems related: [1] [2]. > > I=E2=80=99m not that deep into the authentication modules, but I=E2=80=99= ll take a look > and can keep you posted on this. > Also looping in Eron (in CC) who could perhaps provide more insight on > this at the same time. > > Cheers, > Gordon > > [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 83+-+Allow+multiple+SASL+authenticated+Java+clients+in+ > a+single+JVM+process > [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 85%3A+Dynamic+JAAS+configuration+for+Kafka+clients > > On 26 April 2017 at 8:48:20 PM, Gwenhael Pasquiers ( > gwenhael.pasquiers@ericsson.com) wrote: > > Hello, > > Up to now we=E2=80=99ve been using kafka with jaas (plain login/password)= the > following way: > > - yarnship the jaas file > > - add the jaas file name into =E2=80=9Cflink-conf.yaml=E2=80=9D = using property > =E2=80=9Cenv.java.opts=E2=80=9D > > > > How to support multiple secured kafka 0.10 consumers and producers (with > different logins and password of course) ? > > From what I saw in the kafka sources, the entry name =E2=80=9CKafkaClient= =E2=80=9D is > hardcoded=E2=80=A6 > > Best Regards, > > > > Gwenha=C3=ABl PASQUIERS > > --001a11c017b27052df05505bb289 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Gordon's suggestion seems like a good way to prov= ide per-job credentials based on application-specific properties. =C2=A0 In= contrast, Flink's built-in JAAS features are aimed at making the Flink= cluster's Kerberos credentials available to jobs.

= I want to reiterate that all jobs (for a given Flink cluster) run with full= privilege in the JVM, and that Flink does not guarantee isolation (of secu= rity information) between jobs. =C2=A0 My suggestion is to not run untruste= d job code in a shared Flink cluster.

<= div class=3D"gmail_quote">On Wed, May 24, 2017 at 8:46 PM, Tzu-Li (Gordon) = Tai <tzulitai@apache.org> wrote:
Hi Gwenhael,
Follow-up for this:

Turns out what you re= quire is already available with Kafka 0.10, using dynamic JAAS configuratio= ns [1] instead of a static JAAS file like what you=E2=80=99re currently doi= ng.

The main thing to do is to set a =E2=80=9Csasl.j= aas.config=E2=80=9D in the config properties for your individual Kafka cons= umer / producer.
This will override any static JAAS configuration = used.
Note 2 things here: 1) static JAAS configurations are a JVM = process-wide installation, meaning using that any separate Kafka client wit= hin the same process can always only share the same credentials and 2) the = =E2=80=9CKafkaClient=E2=80=9D is a fixed JAAS lookup section key that the K= afka clients use, which I don=E2=80=99t think is modifiable. So using the s= tatic config approach would never work.

An example = =E2=80=9Csasl.jaas.config=E2=80=9D for plain logins:
"org.apache= .kafka.common.security.plain.PlainLoginModule required username= =3Dxxxx password=3Dyyyy

Simply have different values for= each of the Kafka consumer / producers you=E2=80=99re using.
Cheers,
Gordon


On 8 May 2017 at 4:42:07 PM, Tzu-Li (Gordon) Tai (tzulitai@apache.org) wrote:

Hi Gwenhael,
<= br>
Sorry for the very long delayed response on this.
As you noticed, the =E2=80=9CKafkaClient=E2=80=9D entry name see= ms to be a hardcoded thing on the Kafka side, so currently I don=E2=80=99t = think what you=E2=80=99re asking for is possible.

It= seems like this could be made possible with some of the new authentication= features in Kafka 0.10 that seems related: [1] [2].

I=E2=80=99m not that deep into the authentication modules, but I=E2=80=99l= l take a look and can keep you posted on this.
Also looping in Ero= n (in CC) who could perhaps provide more insight on this at the same time.<= /div>

Cheers,
Gordon

[1]= =C2=A0https://cwiki.apache.org/confluence/display/KAFKA/KIP-= 83+-+Allow+multiple+SASL+authenticated+Java+clients+in+a+sin= gle+JVM+process
[2]=C2=A0https://cwiki.apache.org/confluence/display/KAF= KA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients

On 26 April 2017 at 8:48:20 PM, Gwenhael Pasquiers (= gwenha= el.pasquiers@ericsson.com) wrote:

Hello,

Up to now we=E2=80=99ve been using k= afka with jaas (plain login/password) the following way:

-= =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 yarnship the jaas file

-= =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 add the jaas file name into =E2=80=9Cf= link-conf.yaml=E2=80=9D using property =E2=80=9Cenv.java.opts=E2=80=9D

=C2=A0

How to support multiple secured kafk= a 0.10 consumers and producers (with different logins and password of cours= e) ?

From what I saw in the kafka sources= , the entry name =E2=80=9CKafkaClient=E2=80=9D is hardcoded=E2=80=A6=

Best Regards,

=C2=A0

Gwenha=C3=ABl PASQUIERS


--001a11c017b27052df05505bb289--