kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron Dagostino <rndg...@gmail.com>
Subject Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Date Wed, 15 Jan 2020 18:53:46 GMT
Thanks Colin and Rajini.

I've updated the KIP to say that KIP-421 functionality is available to
encrypt sensitive configs like the ZK key store and trust store
passwords.  (I've also made it clear that the configs are not
dynamically reconfigurable since dynamic values are stored in ZK and
the data is needed to get to ZK in the first place).  Colin, let me
know if you think anything else needs to be said/done about it -- I
wasn't sure if your comment above implies that there are additional
direct actions that we need to take aside from this.

I agree that making the brokers' ZooKeeper clients dynamic with
respect to key and trust stores (e.g. via
zookeeper.ssl.context.supplier.class) may increase the risk that this
KIP may not land in time for 2.5.

Regarding the config names for ZK being different than the ZooKeeper
configs (e.g. camel-case keyStore/trustStore instead of
keystore/truststore, ciphersuites instead of cipher.suites,
enabledProtocols instead of enabled.protocols), I agree it is an
issue.  I tried to mitigate it by putting warning text in the config
docs for these.  Regarding configuration inheritance, I think what you
are saying is that for any configs where the broker has a clear
semantic equivalent, the broker could fall back to the broker value if
the ZK value is not given.  The potential list is:

zookeeper.ssl.keyStore.location => ssl.keystore.location
zookeeper.ssl.keyStore.password => ssl.keystore.password
zookeeper.ssl.keyStore.type => ssl.keystore.type
zookeeper.ssl.trustStore.location => ssl.truststore.location
zookeeper.ssl.trustStore.password => ssl.truststore.password
zookeeper.ssl.trustStore.type => ssl.truststore.type
zookeeper.ssl.ciphersuites => ssl.cipher.suites

Note that there are two configs that look the same based on their key
names but whose allowable values differ:

zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)

Not sure what the right answer is, but hopefully this detail will help
get us there.

Ron

On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cmccabe@apache.org> wrote:
>
> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > Hi Ron,
> >
> > Thanks for the detailed explanation, sounds good to me.
> >
> > A few more questions:
> >
> > 1) At the moment, all sensitive broker configs including
> > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > starting up brokers. We are now adding SSL keystore/truststore passwords
> > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > these to connect to ZK. We should perhaps document that these configs can
> > be encrypted using KIP-421.
>
> That's a good point.  Previously, we didn't have ZK on-the-wire security support.  However,
now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should
be deprecated in favor of using KIP-421.
>
> >
> > 2) We can dynamically update keystores and trust stores used by brokers
> > without restarting the broker. Can we support this easily for ZK clients
> > created by the broker, for example by adding our own
> > `zookeeper.ssl.context.supplier.class`?
> >
>
> Hmm.  That might be better to consider in a follow-up, since the deadline for 2.5 KIPs
is coming up?
>
> best,
> Colin
>
> > 3) Looks like we are using config names that directly map to ZK configs.
> > Have we considered using equivalent Kafka config names with prefixes,
> > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> >
> >
> > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cmccabe@apache.org> wrote:
> >
> > > Hi Ron,
> > >
> > > Thanks for the explanation.  I guess thinking about it a little bit more,
> > > we should just add --zk-tls-config-file to all of these commands.
> > >
> > > We will be removing this option (plus ZK in general) from these commands
> > > in the next major release, but ZK is still supported in 2.5, so we should
> > > just do the logical thing.  (The exception is ZkSecurityMigrator, which
> > > will stay).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > Hi Colin.
> > > >
> > > > <<< It seems like this [--zk-tls-config-file information] could
just
> > > appear
> > > > in a configuration file, which all of these tools already accept (I
> > > think)
> > > >
> > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > "--zk-tls-config-file" parameter is exactly for this purpose.  If we add
> > > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > > (the same code is simply reused; it ends up being just a few extra
> > > lines).
> > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > according
> > > > to the code "This is used only with --bootstrap-server option for
> > > > describing and altering broker configs."
> > > >
> > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > > time
> > > > for the next release.
> > > >
> > > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > special
> > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > > prior to starting Kafka will still exist, so it feels like while
> > > > "--zk-tls-config-file" would never be used except for this use case it
> > > > could probably still be added for this particular situation.
> > > >
> > > > Ron
> > > >
> > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more
I
> > > > sent) did not go through.  I am trying to get emails through now to move
> > > > this discussion forward.
> > > >
> > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmccabe@apache.org>
wrote:
> > > >
> > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > Hi everyone.  I would like to make the following changes to
the KIP.
> > > > > >
> > > > > > MOTIVATION:
> > > > > > Include a statement that it will be difficult in the short term
to
> > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh,
bat}
> > > > > (which
> > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > > cluster
> > > > > > with encrypted passwords in Zookeeper is an explicitly-supported
use
> > > > > case;
> > > > > > therefore it is in scope to be able to securely configure the
CLI
> > > tools
> > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > for TLS
> > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > > zookeeper-security-migration.sh).
> > > > >
> > > > > Hi Ron,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > About deprecations:
> > > > >
> > > > > * zookeeper-security-migration: clearly, deprecating ZK access in
this
> > > one
> > > > > would not make sense, since it would defeat the whole point of the
> > > tool :)
> > > > >
> > > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > > KIP-455
> > > > > implementation has dragged a bit, but this should get done soon.
> > > Certainly
> > > > > before the next release.
> > > > >
> > > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > > I
> > > > > agree there is a super-special bootstrapping case here, but that
should
> > > > > have its own tool, not use kafka-configs.
> > > > >
> > > > > I will post a separate KIP for this, though.
> > > > >
> > > > > >
> > > > > > GOALS:
> > > > > > Support the secure configuration of TLS-encrypted communication
> > > between
> > > > > > Zookeeper and:
> > > > > >   a) Kafka brokers
> > > > > >   b) The three CLI tools mentioned above that still support
direct,
> > > > > > non-deprecated communication to Zookeeper
> > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > communication in CLI tools as part of this KIP; such work will
occur
> > > in
> > > > > > future KIPs instead.
> > > > > >
> > > > > > PUBLIC INTERFACES:
> > > > > > 1) The following new broker configurations will be recognized.
> > > > > >   zookeeper.client.secure (default value = false, for backwards
> > > > > > compatibility)
> > > > > >   zookeeper.clientCnxnSocket
> > > > > >   zookeeper.ssl.keyStore.location
> > > > > >   zookeeper.ssl.keyStore.password
> > > > > >   zookeeper.ssl.trustStore.location
> > > > > >   zookeeper.ssl.trustStore.password
> > > > > > It will be an error for any of the last 5 values to be left
> > > unspecified
> > > > > if
> > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > >
> > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer
class
> > > > > supports
> > > > > > the ability to connect to a different Zookeeper instance than
the
> > > one the
> > > > > > brokers use.  We therefore also add the following optional configs,
> > > which
> > > > > > override the corresponding ones from above when present:
> > > > > >   authorizer.zookeeper.client.secure
> > > > > >   authorizer.zookeeper.clientCnxnSocket
> > > > > >   authorizer.zookeeper.ssl.keyStore.location
> > > > > >   authorizer.zookeeper.ssl.keyStore.password
> > > > > >   authorizer.zookeeper.ssl.trustStore.location
> > > > > >   authorizer.zookeeper.ssl.trustStore.password
> > > > > >
> > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > --zk-tls-config-file
> > > > > > <String: Zookeeper TLS configuration file path>" option.
 The
> > > following
> > > > > > properties will be recognized in that file, and unrecognized
> > > properties
> > > > > > will be ignored to allow the possibility of pointing
> > > zk-tls-config-file
> > > > > at
> > > > > > the broker's config file.
> > > > > >   zookeeper.client.secure (default value = false)
> > > > > >   zookeeper.clientCnxnSocket
> > > > > >   zookeeper.ssl.keyStore.location
> > > > > >   zookeeper.ssl.keyStore.password
> > > > > >   zookeeper.ssl.trustStore.location
> > > > > >   zookeeper.ssl.trustStore.password
> > > > > > It will be an error for any of the last 5 values to be left
> > > unspecified
> > > > > if
> > > > > > zookeeper.client.secure is explicitly set to true.
> > > > >
> > > > > Do we really need a --zk-tls-config-file flag?  It seems like this
> > > could
> > > > > just appear in a configuration file, which all of these tools already
> > > > > accept (I think).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > >
> > > > > > Ron
> > > > > >
> > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndgstn@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi everyone.  Let's get this discussion going again now
that Kafka
> > > 2.4
> > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > >
> > > > > > > First, regarding the KIP number, the other KIP that was
using this
> > > > > number
> > > > > > > moved to KIP 534, so KIP 515 remains the correct number
for this
> > > > > > > discussion.  I've updated the Kafka Improvement Proposal
page to
> > > list
> > > > > this
> > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > >
> > > > > > > Regarding the actual issues under discussion, I think there
are
> > > some
> > > > > > > things we should clarify.
> > > > > > >
> > > > > > > 1) It is possible to use TLS connectivity to Zookeeper
from Apache
> > > > > Kafka
> > > > > > > 2.4 -- the problem is that configuration information has
to be
> > > passed
> > > > > via
> > > > > > > system properties as "-D" command line options on the java
> > > invocation
> > > > > of
> > > > > > > the broker, and those are not secure (anyone with access
to the box
> > > > > can see
> > > > > > > the command line used to invoke the broker); the configuration
> > > includes
> > > > > > > sensitive information (e.g. a keystore password), so we
need a
> > > secure
> > > > > > > mechanism for passing the configuration values.  I believe
the real
> > > > > > > motivation for this KIP is to harden the configuration
mechanism
> > > for
> > > > > > > Zookeeper TLS connectivity.
> > > > > > >
> > > > > > > 2) I believe the list of CLI tools that continue to use
direct
> > > > > Zookeeper
> > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > >   a)
> > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > >   b)
> > > > > > >
> > > > >
> > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > >   c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > >
> > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > > because it
> > > > > > > explicitly states in a comment that a supported use case
is
> > > > > bootstrapping a
> > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > > >
> > > > >
> > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > ).
> > > > > > > This means it will be especially difficult to deprecate
this
> > > particular
> > > > > > > direct Zookeeper connectivity without a different storage
> > > mechanism for
> > > > > > > dynamic configuration values being available (i.e. the
self-managed
> > > > > quorum
> > > > > > > referred to in KIP-500).
> > > > > > >
> > > > > > > I think it would be easier and simpler to harden the Zookeeper
TLS
> > > > > > > configuration in both Kafka and in CLI tools compared to
trying to
> > > > > > > deprecate the direct Zookeeper connectivity in the above
3 CLI
> > > tools --
> > > > > > > especially when ConfigCommand has no obvious short-term
path to
> > > > > deprecation.
> > > > > > >
> > > > > > > Regarding how to harden the TLS configuration, adding the
> > > appropriate
> > > > > > > configs to Kafka is an obvious choice for the brokers.
 Not that
> > > these
> > > > > > > values cannot be dynamically reconfigurable because the
values
> > > would be
> > > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > > >
> > > > > > > Regarding how to harden the TLS configuration for the 3
remaining
> > > CLI
> > > > > > > tools, it would be relatively straightforward to add support
for a
> > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration
system
> > > > > > > properties file path>" option to each one.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Ron
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > pere.urbon@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> True, the number is duplicated.
> > > > > > >>
> > > > > > >>   do you know how can we get the problem fix? I was
not aware, I
> > > > > missed,
> > > > > > >> sorry the need to add the KIP to the table.
> > > > > > >>
> > > > > > >> -- Pere
> > > > > > >>
> > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > quilcate.jorge@gmail.com>
> > > > > del
> > > > > > >> dia
> > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > >>
> > > > > > >> > Hi Pere,
> > > > > > >> >
> > > > > > >> > Have you add your KIP to the list here
> > > > > > >> >
> > > > > > >>
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > >> ?
> > > > > > >> >
> > > > > > >> > I found the KIP number assigned to another.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes
<
> > > > > pere.urbon@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> >> Thanks for your time Harsha,
> > > > > > >> >>    anyone else with comments? looking forward
to hearing from
> > > you.
> > > > > > >> >>
> > > > > > >> >> Stupid question: when do you move from discussion
to vote?
> > > > > > >> >>
> > > > > > >> >> Missatge de Harsha Chintalapani <kafka@harsha.io>
del dia
> > > dv., 30
> > > > > > >> d’ag.
> > > > > > >> >> 2019 a les 21:59:
> > > > > > >> >>
> > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > >> >> > -Harsha
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere
Urbón Bayes <
> > > > > > >> >> pere.urbon@gmail.com>
> > > > > > >> >> > wrote:
> > > > > > >> >> >
> > > > > > >> >> >> Not really,
> > > > > > >> >> >>   my idea is to keep the JAAS parameter,
so people don't see
> > > > > major
> > > > > > >> >> >> changes. But if you pass a properties
file, then this takes
> > > > > > >> precedence
> > > > > > >> >> over
> > > > > > >> >> >> the other, with the idea that you
can do sasl as well with
> > > the
> > > > > > >> >> properties
> > > > > > >> >> >> files.
> > > > > > >> >> >>
> > > > > > >> >> >> Makes sense?
> > > > > > >> >> >>
> > > > > > >> >> >> -- Pere
> > > > > > >> >> >>
> > > > > > >> >> >> Missatge de Harsha Chintalapani <kafka@harsha.io>
del dia
> > > dv.,
> > > > > 30
> > > > > > >> >> d’ag.
> > > > > > >> >> >> 2019 a les 19:00:
> > > > > > >> >> >>
> > > > > > >> >> >>> Hi Pere,
> > > > > > >> >> >>>           Thanks for the KIP.
Enabling SSL for zookeeper
> > > for
> > > > > Kafka
> > > > > > >> >> makes
> > > > > > >> >> >>> sense.
> > > > > > >> >> >>> "The changes are planned to be
introduced in a compatible
> > > way,
> > > > > by
> > > > > > >> >> >>> keeping the current JAAS variable
precedence."
> > > > > > >> >> >>> Can you elaborate a bit here.
If the user configures a JAAS
> > > > > file
> > > > > > >> with
> > > > > > >> >> >>> Client section it will take precedence
over zookeeper SSL
> > > > > configs?
> > > > > > >> >> >>>
> > > > > > >> >> >>> Thanks,
> > > > > > >> >> >>> Harsha
> > > > > > >> >> >>>
> > > > > > >> >> >>>
> > > > > > >> >> >>>
> > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50
AM, Pere Urbón Bayes <
> > > > > > >> >> pere.urbon@gmail.com>
> > > > > > >> >> >>> wrote:
> > > > > > >> >> >>>
> > > > > > >> >> >>>> Hi,
> > > > > > >> >> >>>> quick question, I saw in
another mail that 2.4 release is
> > > > > planned
> > > > > > >> for
> > > > > > >> >> >>>> September. I think it would
be really awesome to have
> > > this for
> > > > > > >> this
> > > > > > >> >> >>>> release, do you think we
can make it?
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> -- Pere
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> Missatge de Pere Urbón Bayes
<pere.urbon@gmail.com> del
> > > dia
> > > > > dj.,
> > > > > > >> 29
> > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> Hi,
> > > > > > >> >> >>>> this is my first KIP for
a change in Apache Kafka, so I'm
> > > > > really
> > > > > > >> need
> > > > > > >> >> >>>> to the process. Looking forward
to hearing from you and
> > > learn
> > > > > the
> > > > > > >> >> best
> > > > > > >> >> >>>> ropes here.
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> I would like to propose this
KIP-515 to enable the
> > > > > > >> ZookeeperClients
> > > > > > >> >> to
> > > > > > >> >> >>>> take full advantage of the
TLS communication in the new
> > > > > Zookeeper
> > > > > > >> >> 3.5.5.
> > > > > > >> >> >>>> Specially interesting it
the Zookeeper Security Migration,
> > > > > that
> > > > > > >> >> without
> > > > > > >> >> >>>> this change will not work
with TLS, disabling users to use
> > > > > ACLs
> > > > > > >> when
> > > > > > >> >> the
> > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> link:
> > > > > > >> >> >>>>
> > > > > > >> >> >>>>
> > > > > > >> >>
> > > > > > >>
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> Looking forward to hearing
from you on this,
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> /cheers
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> --
> > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > >> >> >>>> Software Architect
> > > > > > >> >> >>>> http://www.purbon.com
> > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> --
> > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > >> >> >>>> Software Architect
> > > > > > >> >> >>>> http://www.purbon.com
> > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > >> >> >>>>
> > > > > > >> >> >>>
> > > > > > >> >> >>>
> > > > > > >> >> >>
> > > > > > >> >> >> --
> > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > >> >> >> Software Architect
> > > > > > >> >> >> http://www.purbon.com
> > > > > > >> >> >> https://twitter.com/purbon
> > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >> Pere Urbon-Bayes
> > > > > > >> >> Software Architect
> > > > > > >> >> http://www.purbon.com
> > > > > > >> >> https://twitter.com/purbon
> > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > >> >>
> > > > > > >> >
> > > > > > >>
> > > > > > >> --
> > > > > > >> Pere Urbon-Bayes
> > > > > > >> Software Architect
> > > > > > >> http://www.purbon.com
> > > > > > >> https://twitter.com/purbon
> > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Mime
View raw message