kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism
Date Sun, 12 Jun 2016 21:18:24 GMT
Just to add on that list.

2. It would be good to document the format of the data stored in ZK.
7. Earlier, there was a discussion on whether the tokens should be
propagated through ZK like config/acl/quota, or through the controller.
Currently, the controller is only designed for propagating topic metadata,
but not other data.
8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's
deprecated?

Also, the images in the wiki seem broken.

Thanks,

Jun

On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gwen@confluent.io> wrote:

> From what I can see, remaining questions are:
>
> 1. Who / how are tokens renewed? By original requester only? or using
> Kerberos auth only?
> 2. Are tokens stored on each broker or in ZK?
> 3. How are tokens invalidated / expired?
> 4. Which encryption algorithm is used?
> 5. What is the impersonation proposal (it wasn't in the KIP but was
> discussed in this thread)?
> 6. Do we need new ACLs, if so - for what actions?
>
> Gwen
>
> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <kafka@harsha.io> wrote:
> > Jun & Ismael,
> >                          Unfortunately I couldn't attend the KIP meeting
> >                          when delegation tokens discussed. Appreciate if
> >                          you can update the thread if you have any
> >                          further questions.
> > Thanks,
> > Harsha
> >
> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> >> It seems that the links to images in the KIP are broken.
> >>
> >> Liquan
> >>
> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> >> brahmbhatt.parth@gmail.com> wrote:
> >>
> >> > 110. What does getDelegationTokenAs mean?
> >> > In the current proposal we only allow a user to get delegation token
> for
> >> > the identity that it authenticated as using another mechanism, i.e. A
> user
> >> > that authenticate using a keytab for principal user1@EXAMPLE.COM
> will get
> >> > delegation tokens for that user only. In future I think we will have
> to
> >> > extend support such that we allow some set of users (
> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
> >> > delegation tokens on behalf of other users whose identity they have
> >> > verified independently.  Kafka brokers will have ACLs to control which
> >> > users are allowed to impersonate other users and get tokens on behalf
> of
> >> > them. Overall Impersonation is a whole different problem in my
> opinion and
> >> > I think we can tackle it in separate KIP.
> >> >
> >> > 111. What's the typical rate of getting and renewing delegation
> tokens?
> >> > Typically this should be very very low, 1 request per minute is a
> >> > relatively high estimate. However it depends on the token expiration.
> I am
> >> > less worried about the extra load it puts on controller vs the added
> >> > complexity and the value it offers.
> >> >
> >> > Thanks
> >> > Parth
> >> >
> >> >
> >> >
> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <ismael@juma.me.uk>
> wrote:
> >> >
> >> > > Thanks Rajini. It would probably require a separate KIP as it will
> >> > > introduce user visible changes. We could also update KIP-48 to have
> this
> >> > > information, but it seems cleaner to do it separately. We can
> discuss
> >> > that
> >> > > in the KIP call today.
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> >> > > rajinisivaram@googlemail.com> wrote:
> >> > >
> >> > > > Ismael,
> >> > > >
> >> > > > I have created a JIRA (
> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> >> > > > for adding SCRAM as a SASL mechanism. Would that need another
> KIP? If
> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets
> >> > > reviewed
> >> > > > when the PR is ready?
> >> > > >
> >> > > > Thank you,
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <ismael@juma.me.uk>
> >> > wrote:
> >> > > >
> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> >> > > > >
> >> > > > > Gwen had independently mentioned this as a SASL mechanism that
> might
> >> > be
> >> > > > > useful for Kafka and I have been meaning to check it in more
> detail.
> >> > > Good
> >> > > > > to know that you are willing to contribute an implementation.
> Maybe
> >> > we
> >> > > > > should file a separate JIRA for this?
> >> > > > >
> >> > > > > Ismael
> >> > > > >
> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> >> > > > > rajinisivaram@googlemail.com> wrote:
> >> > > > >
> >> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism) is
> a
> >> > > better
> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in
> SCRAM
> >> > > > > > SaslServer or SaslClient, but I will be happy to add support
> in
> >> > Kafka
> >> > > > > since
> >> > > > > > it would be a useful mechanism to support anyway.
> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol
> for
> >> > > > > > SCRAM-SHA-256.
> >> > > > > >
> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <jun@confluent.io>
> wrote:
> >> > > > > >
> >> > > > > > > Parth,
> >> > > > > > >
> >> > > > > > > Thanks for the explanation. A couple of more questions.
> >> > > > > > >
> >> > > > > > > 110. What does getDelegationTokenAs mean?
> >> > > > > > >
> >> > > > > > > 111. What's the typical rate of getting and renewing
> delegation
> >> > > > tokens?
> >> > > > > > > That may have an impact on whether they should be directed
> to the
> >> > > > > > > controller.
> >> > > > > > >
> >> > > > > > > Jun
> >> > > > > > >
> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >> > > > > > >
> >> > > > > > > > Hi Jun,
> >> > > > > > > >
> >> > > > > > > > Thanks for reviewing.
> >> > > > > > > >
> >> > > > > > > > * We could add a Cluster action to add acls on who can
> request
> >> > > > > > delegation
> >> > > > > > > > tokens. I don't see the use case for that yet but down
> the line
> >> > > > when
> >> > > > > we
> >> > > > > > > > start supporting getDelegationTokenAs it will be
> necessary.
> >> > > > > > > > * Yes we recommend tokens to be only used/distributed over
> >> > secure
> >> > > > > > > channels.
> >> > > > > > > > * Depending on what design we end up choosing
> Invalidation will
> >> > > be
> >> > > > > > > > responsibility of every broker or controller.
> >> > > > > > > > * I am not sure if I documented somewhere that
> invalidation
> >> > will
> >> > > > > > directly
> >> > > > > > > > go through zookeeper but that is not the intent.
> Invalidation
> >> > > will
> >> > > > > > either
> >> > > > > > > > be request based or due to expiration. No direct zookeeper
> >> > > > > interaction
> >> > > > > > > from
> >> > > > > > > > any client.
> >> > > > > > > > * "Broker also stores the DelegationToken without the
> hmac in
> >> > the
> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole purpose
> of
> >> > > > > zookeeper
> >> > > > > > in
> >> > > > > > > > this design is as distribution channel for tokens between
> all
> >> > > > brokers
> >> > > > > > > and a
> >> > > > > > > > layer that ensures only tokens that were generated by
> making a
> >> > > > > request
> >> > > > > > > to a
> >> > > > > > > > broker will be accepted (more on this in second
> paragraph). The
> >> > > > token
> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
> expiration,
> >> > > hmac)
> >> > > > ,
> >> > > > > > one
> >> > > > > > > of
> >> > > > > > > > which is the finally generated hmac but hmac it self is
> >> > derivable
> >> > > > if
> >> > > > > > you
> >> > > > > > > > have all the other elements of the token + secret key to
> >> > generate
> >> > > > > hmac.
> >> > > > > > > > Given zookeeper does not provide SSL support we do not
> want the
> >> > > > > entire
> >> > > > > > > > token to be wire transferred to zookeeper as that will be
> an
> >> > > > insecure
> >> > > > > > > wire
> >> > > > > > > > transfer. Instead we only store all the other elements of
> a
> >> > > > > delegation
> >> > > > > > > > tokens. Brokers can read these elements and because they
> also
> >> > > have
> >> > > > > > access
> >> > > > > > > > to secret key they will be able to generate hmac on their
> end.
> >> > > > > > > >
> >> > > > > > > > One of the alternative proposed is to avoid zookeeper
> >> > > altogether. A
> >> > > > > > > Client
> >> > > > > > > > will call broker with required information (owner, renwer,
> >> > > > > expiration)
> >> > > > > > > and
> >> > > > > > > > get back (signed hmac, uuid). Broker won't store this in
> >> > > zookeeper.
> >> > > > > > From
> >> > > > > > > > this point a client can contact any broker with all the
> >> > > delegation
> >> > > > > > token
> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker
> will
> >> > > > > regenerate
> >> > > > > > > the
> >> > > > > > > > hmac and as long as it matches with hmac presented by
> client ,
> >> > > > broker
> >> > > > > > > will
> >> > > > > > > > allow the request to authenticate.  Only problem with this
> >> > > approach
> >> > > > > is
> >> > > > > > if
> >> > > > > > > > the secret key is compromised any client can now generate
> >> > random
> >> > > > > tokens
> >> > > > > > > and
> >> > > > > > > > they will still be able to authenticate as any user they
> like.
> >> > > with
> >> > > > > > > > zookeeper we guarantee that only tokens acquired via a
> broker
> >> > > > (using
> >> > > > > > some
> >> > > > > > > > auth scheme other than delegation token) will be
> accepted. We
> >> > > need
> >> > > > to
> >> > > > > > > > discuss which proposal makes more sense and we can go
> over it
> >> > in
> >> > > > > > > tomorrow's
> >> > > > > > > > meeting.
> >> > > > > > > >
> >> > > > > > > > Also, can you forward the invite to me?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > Parth
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> jun@confluent.io>
> >> > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks for the KIP. A few comments.
> >> > > > > > > > >
> >> > > > > > > > > 100. This potentially can be useful for Kafka Connect
> and
> >> > Kafka
> >> > > > > rest
> >> > > > > > > > proxy
> >> > > > > > > > > where a worker agent will need to run a task on behalf
> of a
> >> > > > client.
> >> > > > > > We
> >> > > > > > > > will
> >> > > > > > > > > likely need to change how those services use Kafka
> clients
> >> > > > > > > > > (producer/consumer). Instead of a shared client per
> worker,
> >> > we
> >> > > > will
> >> > > > > > > need
> >> > > > > > > > a
> >> > > > > > > > > client per user task since the authentication happens
> at the
> >> > > > > > connection
> >> > > > > > > > > level. For Kafka Connect, the renewer will be the
> workers.
> >> > So,
> >> > > we
> >> > > > > > > > probably
> >> > > > > > > > > need to allow multiple renewers. For Kafka rest proxy,
> the
> >> > > > renewer
> >> > > > > > can
> >> > > > > > > > > probably just be the creator of the token.
> >> > > > > > > > >
> >> > > > > > > > > 101. Do we need new acl on who can request delegation
> tokens?
> >> > > > > > > > >
> >> > > > > > > > > 102. Do we recommend people to send delegation tokens
> in an
> >> > > > > encrypted
> >> > > > > > > > > channel?
> >> > > > > > > > >
> >> > > > > > > > > 103. Who is responsible for expiring tokens, every
> broker?
> >> > > > > > > > >
> >> > > > > > > > > 104. For invalidating tokens, would it be better to do
> it in
> >> > a
> >> > > > > > request
> >> > > > > > > > > instead of going to ZK directly?
> >> > > > > > > > >
> >> > > > > > > > > 105. The terminology of client in the wiki sometimes
> refers
> >> > to
> >> > > > the
> >> > > > > > end
> >> > > > > > > > > client and some other times refers to the client using
> the
> >> > > > > delegation
> >> > > > > > > > > tokens. It would be useful to distinguish between the
> two.
> >> > > > > > > > >
> >> > > > > > > > > 106. Could you explain the sentence "Broker also stores
> the
> >> > > > > > > > DelegationToken
> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I
> thought the
> >> > > > > > > delegation
> >> > > > > > > > > token is the hmac.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Jun
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> jun@confluent.io>
> >> > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi, Harsha,
> >> > > > > > > > > >
> >> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss
> this in
> >> > > the
> >> > > > > > > meeting
> >> > > > > > > > > > tomorrow.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > Jun
> >> > > > > > > > > >
> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> kafka@harsha.io>
> >> > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > >> Hi All,
> >> > > > > > > > > >>            Can we have a KIP meeting around this.
> The KIP
> >> > is
> >> > > > up
> >> > > > > > for
> >> > > > > > > > > >>            sometime and if there are any questions
> lets
> >> > > > quickly
> >> > > > > > hash
> >> > > > > > > > out
> >> > > > > > > > > >>            details.
> >> > > > > > > > > >>
> >> > > > > > > > > >> Thanks,
> >> > > > > > > > > >> Harsha
> >> > > > > > > > > >>
> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt
> wrote:
> >> > > > > > > > > >> > That is what the hadoop echo system uses so no good
> >> > reason
> >> > > > > > really.
> >> > > > > > > > We
> >> > > > > > > > > >> > could
> >> > > > > > > > > >> > change it to whatever is the newest recommended
> standard
> >> > > is.
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > Thanks
> >> > > > > > > > > >> > Parth
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> >> > > > > ismael@juma.me.uk
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > > Hi Parth,
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing
> this and
> >> > > may
> >> > > > > have
> >> > > > > > > > > >> additional
> >> > > > > > > > > >> > > questions later. The immediate question that
> came to
> >> > > mind
> >> > > > is
> >> > > > > > our
> >> > > > > > > > > >> choice of
> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE
> in
> >> > the
> >> > > > IANA
> >> > > > > > > > > Registry
> >> > > > > > > > > >> of
> >> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has
> been
> >> > > moved
> >> > > > > to
> >> > > > > > > > > Historic
> >> > > > > > > > > >> > > status:
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> >> > > > > > > > > >> > >
> >> > > > > > > > >
> >> > > > > >
> >> > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > What is the reasoning behind that choice?
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > Thanks,
> >> > > > > > > > > >> > > Ismael
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> >> > > > > > > gwen@confluent.io
> >> > > > > > > > >
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > > Also comments inline :)
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * I want to emphasize that even though
> delegation
> >> > > > tokens
> >> > > > > > > are a
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > > innovation, I feel very strongly about not
> adding
> >> > > > > > dependency
> >> > > > > > > > on
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > > when implementing delegation tokens for
> Kafka. The
> >> > > KIP
> >> > > > > > > doesn't
> >> > > > > > > > > >> imply
> >> > > > > > > > > >> > > > > such dependency, but if you can clarify...
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will
> read
> >> > the
> >> > > > KIP
> >> > > > > > and
> >> > > > > > > > > panic
> >> > > > > > > > > >> > > > three weeks before the next release...
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * Can we get delegation token at any time
> after
> >> > > > > > > > authenticating?
> >> > > > > > > > > >> only
> >> > > > > > > > > >> > > > > immediately after?
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *As long as you are authenticated you can get
> >> > > > delegation
> >> > > > > > > > tokens.
> >> > > > > > > > > >> We
> >> > > > > > > > > >> > > need
> >> > > > > > > > > >> > > > to
> >> > > > > > > > > >> > > > > discuss if a client authenticated using
> delegation
> >> > > > > token,
> >> > > > > > > can
> >> > > > > > > > > also
> >> > > > > > > > > >> > > > acquire
> >> > > > > > > > > >> > > > > delegation token again or not. Also there is
> the
> >> > > > > question
> >> > > > > > of
> >> > > > > > > > do
> >> > > > > > > > > we
> >> > > > > > > > > >> > > allow
> >> > > > > > > > > >> > > > > anyone to acquire delegation token or we want
> >> > > specific
> >> > > > > > ACLs
> >> > > > > > > (I
> >> > > > > > > > > >> think
> >> > > > > > > > > >> > > its
> >> > > > > > > > > >> > > > an
> >> > > > > > > > > >> > > > > overkill.)*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > I think we are debating two options: Either
> require
> >> > > > > Kerberos
> >> > > > > > > > auth
> >> > > > > > > > > >> for
> >> > > > > > > > > >> > > > renewal or require non-owners to renew.
> >> > > > > > > > > >> > > > I *think* the latter is simpler (it basically
> >> > require
> >> > > a
> >> > > > > "job
> >> > > > > > > > > master"
> >> > > > > > > > > >> > > > to take responsibility for the renewal, it
> will have
> >> > > its
> >> > > > > own
> >> > > > > > > > > >> identity
> >> > > > > > > > > >> > > > anyway and I think this is the correct design
> >> > pattern
> >> > > > > > anyway.
> >> > > > > > > > For
> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
> renewals?),
> >> > but
> >> > > > it
> >> > > > > is
> >> > > > > > > > hard
> >> > > > > > > > > to
> >> > > > > > > > > >> > > > debate simplicity without looking at the code
> >> > changes
> >> > > > > > > required.
> >> > > > > > > > If
> >> > > > > > > > > >> you
> >> > > > > > > > > >> > > > have a draft of how the "require Kerberos"
> will look
> >> > > in
> >> > > > > > Kafka
> >> > > > > > > > > code,
> >> > > > > > > > > >> > > > I'll be happy to take a look.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * My understanding is that tokens will
> propagate
> >> > via
> >> > > > ZK
> >> > > > > > but
> >> > > > > > > > > >> without
> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
> protocol,
> >> > > > correct?
> >> > > > > > > > Clients
> >> > > > > > > > > >> > > > > currently don't retry on SASL auth failure
> (IIRC),
> >> > > but
> >> > > > > > since
> >> > > > > > > > the
> >> > > > > > > > > >> > > > > tokens propagate between brokers asynch, we
> will
> >> > > need
> >> > > > to
> >> > > > > > > > retry a
> >> > > > > > > > > >> bit
> >> > > > > > > > > >> > > > > to avoid clients failing auth due to timing
> >> > issues.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right now.
> The
> >> > > > current
> >> > > > > > > > > documented
> >> > > > > > > > > >> > > > approach
> >> > > > > > > > > >> > > > > is zookeeper based and it does not require
> any
> >> > > changes
> >> > > > > to
> >> > > > > > > > > >> > > UpdateMetadata
> >> > > > > > > > > >> > > > > protocol. An alternative approach can remove
> >> > > zookeeper
> >> > > > > > > > > dependency
> >> > > > > > > > > >> as
> >> > > > > > > > > >> > > well
> >> > > > > > > > > >> > > > > but we can discuss that in KIP discussion
> call.*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping
> Jun to
> >> > > > > > arrange a
> >> > > > > > > > > call?
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having just
> the
> >> > > > > > controller
> >> > > > > > > > > issue
> >> > > > > > > > > >> the
> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared
> >> > secret.
> >> > > > Not
> >> > > > > > > sure
> >> > > > > > > > if
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > > > want to continue the discussion here or on
> the
> >> > > wiki. I
> >> > > > > > think
> >> > > > > > > > > that
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > > > can decouple the problem of "token
> distribution"
> >> > > from
> >> > > > > > > "shared
> >> > > > > > > > > >> secret
> >> > > > > > > > > >> > > > > distribution" and use the controller as the
> only
> >> > > token
> >> > > > > > > > generator
> >> > > > > > > > > >> to
> >> > > > > > > > > >> > > > > solve the second issue, while still using ZK
> async
> >> > > to
> >> > > > > > > > distribute
> >> > > > > > > > > >> > > > > tokens.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I am
> fine with
> >> > > > that
> >> > > > > > > > approach
> >> > > > > > > > > >> as
> >> > > > > > > > > >> > > long
> >> > > > > > > > > >> > > > as
> >> > > > > > > > > >> > > > > we agree that the extra complexity of
> >> > > adding/updating
> >> > > > > APIS
> >> > > > > > > > adds
> >> > > > > > > > > >> enough
> >> > > > > > > > > >> > > > > value. The advantage with the controller
> approach
> >> > is
> >> > > > > > secret
> >> > > > > > > > > >> rotation
> >> > > > > > > > > >> > > can
> >> > > > > > > > > >> > > > be
> >> > > > > > > > > >> > > > > automated,frequent and would not require
> >> > > deployment. *
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Can you detail the extra complexity (or point
> me to
> >> > > the
> >> > > > > > email
> >> > > > > > > I
> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> >> > > > > > > > > >> > > > As far as I can tell, clients can already find
> the
> >> > > > > > controller
> >> > > > > > > > from
> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about
> controller
> >> > > > load.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > * While I like the idea of forcing kerberos
> auth
> >> > for
> >> > > > > > > renwal, I
> >> > > > > > > > > >> think
> >> > > > > > > > > >> > > > > it mixes the transport layer the the request
> >> > content
> >> > > > in
> >> > > > > a
> >> > > > > > > > pretty
> >> > > > > > > > > >> ugly
> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is
> >> > > better.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. While this
> will
> >> > > make
> >> > > > > the
> >> > > > > > > > kafka
> >> > > > > > > > > >> code
> >> > > > > > > > > >> > > > > pretty straight forward , forcing  renewer to
> >> > > > non-owner
> >> > > > > > > pushes
> >> > > > > > > > > >> the code
> >> > > > > > > > > >> > > > > ugliness to client and makes it even harder
> to
> >> > > > > > integrate.  *
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > As mentioned before, I don't think the
> "renewal by
> >> > > > other"
> >> > > > > > > > approach
> >> > > > > > > > > >> is
> >> > > > > > > > > >> > > > that ugly for the clients we expect to use
> >> > delegation
> >> > > > > tokens
> >> > > > > > > > since
> >> > > > > > > > > >> > > > they will have an app-master of some sort who
> >> > > requested
> >> > > > > the
> >> > > > > > > > token
> >> > > > > > > > > to
> >> > > > > > > > > >> > > > begin with.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > The response for my question on how multiple
> >> > > > identities
> >> > > > > > will
> >> > > > > > > > be
> >> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we
> don't
> >> > > > > > > > authenticate
> >> > > > > > > > > >> each
> >> > > > > > > > > >> > > > > request, we authenticate connections.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *We authenticate connections, and only when
> they
> >> > are
> >> > > > > being
> >> > > > > > > > > >> established.
> >> > > > > > > > > >> > > > Let
> >> > > > > > > > > >> > > > > me try to phrase this as a question, in
> absence of
> >> > > > > > > delegation
> >> > > > > > > > > >> tokens if
> >> > > > > > > > > >> > > > we
> >> > > > > > > > > >> > > > > had to support the use case using user TGT's
> how
> >> > > would
> >> > > > > we
> >> > > > > > do
> >> > > > > > > > it?
> >> > > > > > > > > >> My
> >> > > > > > > > > >> > > point
> >> > > > > > > > > >> > > > > was it would be no different with delegation
> >> > tokens.
> >> > > > The
> >> > > > > > use
> >> > > > > > > > > case
> >> > > > > > > > > >> you
> >> > > > > > > > > >> > > are
> >> > > > > > > > > >> > > > > describing seems more like impersonation.*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Yeah, I thought that one of the things that
> >> > delegation
> >> > > > > > tokens
> >> > > > > > > > > >> handled.
> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Gwen
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > Thanks
> >> > > > > > > > > >> > > > > Parth
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen
> Shapira <
> >> > > > > > > > > gwen@confluent.io
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> Few more comments:
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * The API / RequestResponse section doesn't
> seem
> >> > to
> >> > > > > have
> >> > > > > > > good
> >> > > > > > > > > >> > > > >> description of the changes to the Kafka
> Protocol.
> >> > > > > Sounds
> >> > > > > > > like
> >> > > > > > > > > >> you are
> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
> >> > > > > > RenewTokenRequest
> >> > > > > > > > (and
> >> > > > > > > > > >> > > > >> matching responses), without detailing the
> >> > contents
> >> > > > of
> >> > > > > > the
> >> > > > > > > > > >> requests
> >> > > > > > > > > >> > > > >> and responses? Or rather, you show the class
> >> > > > interface,
> >> > > > > > but
> >> > > > > > > > not
> >> > > > > > > > > >> the
> >> > > > > > > > > >> > > > >> underlying protocol. This could be seen as
> an
> >> > > > > > > implementation
> >> > > > > > > > > >> detail,
> >> > > > > > > > > >> > > > >> but since the binary protocol is what we
> provide
> >> > to
> >> > > > > > > non-Java
> >> > > > > > > > > >> clients,
> >> > > > > > > > > >> > > > >> we need to show the changes there.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> >> > > > > > > > delegationTokenRequestHandler?
> >> > > > > > > > > >> Is it
> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client?
> Its
> >> > > > > unclear...
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * I want to emphasize that even though
> delegation
> >> > > > > tokens
> >> > > > > > > are
> >> > > > > > > > a
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > >> innovation, I feel very strongly about not
> adding
> >> > > > > > > dependency
> >> > > > > > > > on
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > >> when implementing delegation tokens for
> Kafka.
> >> > The
> >> > > > KIP
> >> > > > > > > > doesn't
> >> > > > > > > > > >> imply
> >> > > > > > > > > >> > > > >> such dependency, but if you can clarify...
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * Can we get delegation token at any time
> after
> >> > > > > > > > authenticating?
> >> > > > > > > > > >> only
> >> > > > > > > > > >> > > > >> immediately after?
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * My understanding is that tokens will
> propagate
> >> > > via
> >> > > > ZK
> >> > > > > > but
> >> > > > > > > > > >> without
> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata
> protocol,
> >> > > > correct?
> >> > > > > > > > Clients
> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth failure
> >> > (IIRC),
> >> > > > but
> >> > > > > > > since
> >> > > > > > > > > the
> >> > > > > > > > > >> > > > >> tokens propagate between brokers asynch, we
> will
> >> > > need
> >> > > > > to
> >> > > > > > > > retry
> >> > > > > > > > > a
> >> > > > > > > > > >> bit
> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to timing
> >> > issues.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not touching
> ZK
> >> > > > directly
> >> > > > > > :)
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having
> just the
> >> > > > > > controller
> >> > > > > > > > > >> issue the
> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared
> >> > > secret.
> >> > > > > Not
> >> > > > > > > sure
> >> > > > > > > > > if
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > > >> want to continue the discussion here or on
> the
> >> > > wiki.
> >> > > > I
> >> > > > > > > think
> >> > > > > > > > > >> that we
> >> > > > > > > > > >> > > > >> can decouple the problem of "token
> distribution"
> >> > > from
> >> > > > > > > "shared
> >> > > > > > > > > >> secret
> >> > > > > > > > > >> > > > >> distribution" and use the controller as the
> only
> >> > > > token
> >> > > > > > > > > generator
> >> > > > > > > > > >> to
> >> > > > > > > > > >> > > > >> solve the second issue, while still using ZK
> >> > async
> >> > > to
> >> > > > > > > > > distribute
> >> > > > > > > > > >> > > > >> tokens.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite
> lifetime
> >> > of
> >> > > > > > tokens
> >> > > > > > > > (and
> >> > > > > > > > > >> hoped
> >> > > > > > > > > >> > > > >> to hear from others in the community) - but
> >> > having
> >> > > > the
> >> > > > > > > option
> >> > > > > > > > > and
> >> > > > > > > > > >> > > > >> documenting it as less secure, allows users
> to
> >> > > > > configure
> >> > > > > > > > their
> >> > > > > > > > > >> system
> >> > > > > > > > > >> > > > >> as the wish.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * While I like the idea of forcing kerberos
> auth
> >> > > for
> >> > > > > > > renwal,
> >> > > > > > > > I
> >> > > > > > > > > >> think
> >> > > > > > > > > >> > > > >> it mixes the transport layer the the request
> >> > > content
> >> > > > > in a
> >> > > > > > > > > pretty
> >> > > > > > > > > >> ugly
> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner
> is
> >> > > better.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * More detailed explanation on what we plan
> to do
> >> > > for
> >> > > > > the
> >> > > > > > > > java
> >> > > > > > > > > >> clients
> >> > > > > > > > > >> > > > >> specifically - new configuration? new APIs?
> >> > > > > > > > > >> > > > >> The response for my question on how multiple
> >> > > > identities
> >> > > > > > > will
> >> > > > > > > > be
> >> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we
> >> > don't
> >> > > > > > > > authenticate
> >> > > > > > > > > >> each
> >> > > > > > > > > >> > > > >> request, we authenticate connections.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only
> used
> >> > in
> >> > > > the
> >> > > > > > > Hadoop
> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> >> > alternatives
> >> > > in
> >> > > > > > other
> >> > > > > > > > > >> ecosystems
> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether
> there
> >> > are
> >> > > > some
> >> > > > > > > > > >> advantages
> >> > > > > > > > > >> > > > >> there.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> Gwen
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> >> > > > > kafka@harsha.io
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >> > > > >> > Hi Gwen,
> >> > > > > > > > > >> > > > >> >            Can you look at Parth's last
> reply.
> >> > > Does
> >> > > > > it
> >> > > > > > > > answer
> >> > > > > > > > > >> your
> >> > > > > > > > > >> > > > >> >            concerns.
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> > Thanks,
> >> > > > > > > > > >> > > > >> > Harsha
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
> >> > > brahmbhatt
> >> > > > > > wrote:
> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki
> already
> >> > has
> >> > > > > > details
> >> > > > > > > on
> >> > > > > > > > > >> token
> >> > > > > > > > > >> > > > >> >> expiration
> >> > > > > > > > > >> > > > >> >> under token acquisition process
> >> > > > > > > > > >> > > > >> >> <
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >> > > > > > > > > >> > > > >> >.
> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens will
> expire
> >> > > based
> >> > > > > on a
> >> > > > > > > > > server
> >> > > > > > > > > >> side
> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless
> >> > renewed.
> >> > > > > > Renewal
> >> > > > > > > > is
> >> > > > > > > > > >> only
> >> > > > > > > > > >> > > > allowed
> >> > > > > > > > > >> > > > >> >> until the max life time of token.
> >> > Alternatively
> >> > > we
> >> > > > > > could
> >> > > > > > > > > also
> >> > > > > > > > > >> make
> >> > > > > > > > > >> > > > that
> >> > > > > > > > > >> > > > >> >> an
> >> > > > > > > > > >> > > > >> >> optional param and the server side
> default can
> >> > > > serve
> >> > > > > > as
> >> > > > > > > > the
> >> > > > > > > > > >> upper
> >> > > > > > > > > >> > > > bound.
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> To your second point it will be done
> exactly
> >> > the
> >> > > > > same
> >> > > > > > > way
> >> > > > > > > > we
> >> > > > > > > > > >> would
> >> > > > > > > > > >> > > > >> >> support
> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client
> will have
> >> > > to
> >> > > > > put
> >> > > > > > > the
> >> > > > > > > > > >> tokens it
> >> > > > > > > > > >> > > > >> wants
> >> > > > > > > > > >> > > > >> >> to use in the subject instance and call
> >> > > > > > produce/consume
> >> > > > > > > > > inside
> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to
> keep
> >> > > track
> >> > > > of
> >> > > > > > its
> >> > > > > > > > own
> >> > > > > > > > > >> > > > subject. I
> >> > > > > > > > > >> > > > >> >> will have to look at the code to see if
> we
> >> > > support
> >> > > > > > this
> >> > > > > > > > > >> feature
> >> > > > > > > > > >> > > right
> >> > > > > > > > > >> > > > >> now
> >> > > > > > > > > >> > > > >> >> but my understanding is delegation token
> >> > > shouldn't
> >> > > > > > need
> >> > > > > > > > any
> >> > > > > > > > > >> special
> >> > > > > > > > > >> > > > >> >> treatment as its just another type of
> >> > Credential
> >> > > > in
> >> > > > > > the
> >> > > > > > > > > >> subject.
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> I would also like to know what is your
> opinion
> >> > > > about
> >> > > > > > > > > infinite
> >> > > > > > > > > >> > > renewal
> >> > > > > > > > > >> > > > >> (my
> >> > > > > > > > > >> > > > >> >> recommendation is to not support this),
> tokens
> >> > > > > > renewing
> >> > > > > > > > them
> >> > > > > > > > > >> > > self(my
> >> > > > > > > > > >> > > > >> >> recommendation is to not support this)
> and
> >> > most
> >> > > > > > > > importantly
> >> > > > > > > > > >> your
> >> > > > > > > > > >> > > > choice
> >> > > > > > > > > >> > > > >> >> between the alternatives listed on this
> thread
> >> > > > > > > > > >> > > > >> >> <
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2
> minus
> >> > > > > > > controller
> >> > > > > > > > > >> > > > distributing
> >> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> Thanks
> >> > > > > > > > > >> > > > >> >> Parth
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen
> Shapira <
> >> > > > > > > > > >> gwen@confluent.io>
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> > Harsha,
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I
> didn't
> >> > see
> >> > > > > your
> >> > > > > > > > design
> >> > > > > > > > > >> yet,
> >> > > > > > > > > >> > > > but in
> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of producers,
> which
> >> > > > will
> >> > > > > > > serve
> >> > > > > > > > > >> multiple
> >> > > > > > > > > >> > > > users
> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since these
> users
> >> > > will
> >> > > > > have
> >> > > > > > > > > >> different
> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> authenticate
> >> > > > > separately,
> >> > > > > > > and
> >> > > > > > > > > >> can't
> >> > > > > > > > > >> > > > share a
> >> > > > > > > > > >> > > > >> >> > token.
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > Gwen
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha
> <
> >> > > > > > > kafka@harsha.io
> >> > > > > > > > >
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > > > >> >> > > Gwen,
> >> > > > > > > > > >> > > > >> >> > >            On your second point. Can
> you
> >> > > > > describe
> >> > > > > > a
> >> > > > > > > > > >> usecase
> >> > > > > > > > > >> > > where
> >> > > > > > > > > >> > > > >> >> > >            mutliple clients ended up
> >> > > sharing a
> >> > > > > > > > producer
> >> > > > > > > > > >> and
> >> > > > > > > > > >> > > even
> >> > > > > > > > > >> > > > if
> >> > > > > > > > > >> > > > >> they
> >> > > > > > > > > >> > > > >> >> > >            do why can't they not use
> >> > single
> >> > > > > token
> >> > > > > > > that
> >> > > > > > > > > >> producer
> >> > > > > > > > > >> > > > >> >> > >            captures. Why would we
> need
> >> > > > multiple
> >> > > > > > > > clients
> >> > > > > > > > > >> with
> >> > > > > > > > > >> > > > >> different
> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a single
> >> > instance
> >> > > of
> >> > > > > > > > producer.
> >> > > > > > > > > >> Also
> >> > > > > > > > > >> > > in
> >> > > > > > > > > >> > > > >> this
> >> > > > > > > > > >> > > > >> >> > >            case other clients have
> access
> >> > > all
> >> > > > > the
> >> > > > > > > > tokens
> >> > > > > > > > > >> no?
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > > Thanks,
> >> > > > > > > > > >> > > > >> >> > > Harsha
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM,
> Gwen
> >> > > Shapira
> >> > > > > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> >> > > > > > > > > >> > > > >> >> > >>
> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see in
> the
> >> > > wiki:
> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for
> delegation
> >> > > > > tokens?
> >> > > > > > > > > >> Renewal? How
> >> > > > > > > > > >> > > > do we
> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation
> tokens
> >> > for
> >> > > > > > "do-as"
> >> > > > > > > > > (say,
> >> > > > > > > > > >> > > submit
> >> > > > > > > > > >> > > > >> Storm
> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a
> producer
> >> > > for
> >> > > > > > every
> >> > > > > > > > job
> >> > > > > > > > > >> (we
> >> > > > > > > > > >> > > can't
> >> > > > > > > > > >> > > > >> share
> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs running
> on
> >> > same
> >> > > > > node),
> >> > > > > > > > since
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > only
> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is
> there a
> >> > > plan
> >> > > > > to
> >> > > > > > > > change
> >> > > > > > > > > >> this
> >> > > > > > > > > >> > > for
> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow
> >> > > multiple
> >> > > > > > users
> >> > > > > > > > > with
> >> > > > > > > > > >> > > > different
> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> >> > > > > > > > > >> > > > >> >> > >>
> >> > > > > > > > > >> > > > >> >> > >> Gwen
> >> > > > > > > > > >> > > > >> >> > >>
> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM,
> parth
> >> > > > > brahmbhatt
> >> > > > > > > > > >> > > > >> >> > >> <brahmbhatt.parth@gmail.com> wrote:
> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can
> >> > other
> >> > > > > > > committers
> >> > > > > > > > > >> review?
> >> > > > > > > > > >> > > > >> >> > >> >
> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> >> > > > > > > > > >> > > > >> >> > >> > Parth
> >> > > > > > > > > >> > > > >> >> > >> >
> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM,
> >> > Harsha <
> >> > > > > > > > > >> kafka@harsha.io>
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> >> > > > > > > > > >> > > > >> >> > >> >>           Overall current design
> looks
> >> > > > good
> >> > > > > to
> >> > > > > > > > me. I
> >> > > > > > > > > >> am +1
> >> > > > > > > > > >> > > on
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > KIP.
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as
> >> > well.
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57
> AM,
> >> > parth
> >> > > > > > > > brahmbhatt
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of
> infinite
> >> > > > lifetime
> >> > > > > > > but I
> >> > > > > > > > > >> see the
> >> > > > > > > > > >> > > > >> Streaming
> >> > > > > > > > > >> > > > >> >> > use
> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use
> case I
> >> > > was
> >> > > > > > hoping
> >> > > > > > > > > >> there will
> >> > > > > > > > > >> > > > be
> >> > > > > > > > > >> > > > >> some
> >> > > > > > > > > >> > > > >> >> > notion
> >> > > > > > > > > >> > > > >> >> > >> >> > of
> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
> >> > > > delegation
> >> > > > > > > tokens
> >> > > > > > > > > at
> >> > > > > > > > > >> fixed
> >> > > > > > > > > >> > > > >> interval
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that
> is
> >> > not
> >> > > > the
> >> > > > > > case
> >> > > > > > > > for
> >> > > > > > > > > >> we can
> >> > > > > > > > > >> > > > >> discuss
> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing
> them self
> >> > > and
> >> > > > > the
> >> > > > > > > > > >> security
> >> > > > > > > > > >> > > > >> implications
> >> > > > > > > > > >> > > > >> >> > of the
> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch
> >> > tokens
> >> > > > > from
> >> > > > > > > > > >> zookeeper,
> >> > > > > > > > > >> > > > >> overall I
> >> > > > > > > > > >> > > > >> >> > think
> >> > > > > > > > > >> > > > >> >> > >> >> > its
> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely
> on our
> >> > > > > metadata
> >> > > > > > > > store
> >> > > > > > > > > >> and I
> >> > > > > > > > > >> > > > >> think we
> >> > > > > > > > > >> > > > >> >> > are
> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with
> all
> >> > the
> >> > > > > KIP-4
> >> > > > > > > > > >> improvements.
> >> > > > > > > > > >> > > > I
> >> > > > > > > > > >> > > > >> chose
> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the
> client
> >> > > > will
> >> > > > > > > still
> >> > > > > > > > > >> just talk
> >> > > > > > > > > >> > > > to
> >> > > > > > > > > >> > > > >> >> > broker , its
> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use
> zookeeper
> >> > > > which
> >> > > > > we
> >> > > > > > > > > >> already do
> >> > > > > > > > > >> > > > for a
> >> > > > > > > > > >> > > > >> lot
> >> > > > > > > > > >> > > > >> >> > of
> >> > > > > > > > > >> > > > >> >> > >> >> > other
> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of development
> + and
> >> > > the
> >> > > > > > > ability
> >> > > > > > > > > so
> >> > > > > > > > > >> > > tokens
> >> > > > > > > > > >> > > > >> will
> >> > > > > > > > > >> > > > >> >> > survive
> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
> >> > > failure.
> >> > > > > if a
> >> > > > > > > > > >> majority
> >> > > > > > > > > >> > > > agrees
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > added
> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller
> >> > > forwarding
> >> > > > > > keys
> >> > > > > > > to
> >> > > > > > > > > all
> >> > > > > > > > > >> > > > broker is
> >> > > > > > > > > >> > > > >> >> > justified
> >> > > > > > > > > >> > > > >> >> > >> >> > as
> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security ,
> I am
> >> > > fine
> >> > > > > > with
> >> > > > > > > > that
> >> > > > > > > > > >> option
> >> > > > > > > > > >> > > > too.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not
> support SSL
> >> > > we
> >> > > > > can
> >> > > > > > > not
> >> > > > > > > > > >> store
> >> > > > > > > > > >> > > > master
> >> > > > > > > > > >> > > > >> keys
> >> > > > > > > > > >> > > > >> >> > in
> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will
> be
> >> > > exposed
> >> > > > > on
> >> > > > > > > > wire.
> >> > > > > > > > > To
> >> > > > > > > > > >> > > > support
> >> > > > > > > > > >> > > > >> >> > rotation
> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting current
> clients is
> >> > > > > > > something I
> >> > > > > > > > > >> need to
> >> > > > > > > > > >> > > > put
> >> > > > > > > > > >> > > > >> more
> >> > > > > > > > > >> > > > >> >> > thought
> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal
> assumes the
> >> > > > > rotation
> >> > > > > > > > will
> >> > > > > > > > > >> > > > invalidate
> >> > > > > > > > > >> > > > >> all
> >> > > > > > > > > >> > > > >> >> > current
> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also
> review
> >> > > and
> >> > > > > post
> >> > > > > > > > their
> >> > > > > > > > > >> > > comments
> >> > > > > > > > > >> > > > >> so we
> >> > > > > > > > > >> > > > >> >> > can
> >> > > > > > > > > >> > > > >> >> > >> >> > make
> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39
> AM,
> >> > > Ashish
> >> > > > > > Singh
> >> > > > > > > <
> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at
> 11:26 AM,
> >> > > > > Harsha <
> >> > > > > > > > > >> > > > kafka@harsha.io>
> >> > > > > > > > > >> > > > >> >> > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion
> >> > > threads
> >> > > > on
> >> > > > > > > this
> >> > > > > > > > > KIP.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from
> >> > Jitendra
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large
> number of
> >> > > > > clients
> >> > > > > > > that
> >> > > > > > > > > are
> >> > > > > > > > > >> > > > running
> >> > > > > > > > > >> > > > >> all
> >> > > > > > > > > >> > > > >> >> > over the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate
> with
> >> > > Kafka
> >> > > > > > > > brokers,
> >> > > > > > > > > >> is very
> >> > > > > > > > > >> > > > >> similar
> >> > > > > > > > > >> > > > >> >> > to the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large
> number
> >> > of
> >> > > > > tasks
> >> > > > > > > > > running
> >> > > > > > > > > >> > > across
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > cluster
> >> > > > > > > > > >> > > > >> >> > >> >> that
> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
> >> > > > Namenode.
> >> > > > > > > > > >> Therefore, the
> >> > > > > > > > > >> > > > >> >> > delegation token
> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a
> good
> >> > fit
> >> > > > for
> >> > > > > > > this
> >> > > > > > > > > use
> >> > > > > > > > > >> case
> >> > > > > > > > > >> > > > as we
> >> > > > > > > > > >> > > > >> >> > have seen
> >> > > > > > > > > >> > > > >> >> > >> >> it
> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in
> HDFS
> >> > and
> >> > > > > YARN.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is
> very
> >> > much
> >> > > > > > inline
> >> > > > > > > > with
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > >> >> > approach. A few
> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to
> allow
> >> > > > > > infinite
> >> > > > > > > > > >> renewable
> >> > > > > > > > > >> > > > >> lifetime
> >> > > > > > > > > >> > > > >> >> > for a
> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a
> token
> >> > to a
> >> > > > max
> >> > > > > > > life
> >> > > > > > > > > time
> >> > > > > > > > > >> > > > (default
> >> > > > > > > > > >> > > > >> 7
> >> > > > > > > > > >> > > > >> >> > days).  A
> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
> >> > believed
> >> > > to
> >> > > > > > > > increase
> >> > > > > > > > > >> with
> >> > > > > > > > > >> > > > time.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
> >> > > lifetime
> >> > > > > > might
> >> > > > > > > > not
> >> > > > > > > > > >> be the
> >> > > > > > > > > >> > > > best
> >> > > > > > > > > >> > > > >> idea.
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the
> tokens
> >> > are
> >> > > > > stored
> >> > > > > > > in
> >> > > > > > > > > >> zookeeper
> >> > > > > > > > > >> > > > as
> >> > > > > > > > > >> > > > >> well,
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> can
> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is
> clever
> >> > > as
> >> > > > it
> >> > > > > > can
> >> > > > > > > > > allow
> >> > > > > > > > > >> > > > >> replacing the
> >> > > > > > > > > >> > > > >> >> > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max
> life
> >> > > time,
> >> > > > > and
> >> > > > > > > > > clients
> >> > > > > > > > > >> can
> >> > > > > > > > > >> > > > >> download
> >> > > > > > > > > >> > > > >> >> > new
> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
> shouldn't be
> >> > a
> >> > > > big
> >> > > > > > load
> >> > > > > > > > on
> >> > > > > > > > > >> > > zookeeper
> >> > > > > > > > > >> > > > >> as a
> >> > > > > > > > > >> > > > >> >> > client
> >> > > > > > > > > >> > > > >> >> > >> >> will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token
> once in
> >> > > > > several
> >> > > > > > > > days.
> >> > > > > > > > > >> In this
> >> > > > > > > > > >> > > > >> approach
> >> > > > > > > > > >> > > > >> >> > you
> >> > > > > > > > > >> > > > >> >> > >> >> don't
> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on
> the
> >> > > token
> >> > > > > even
> >> > > > > > > for
> >> > > > > > > > > >> long
> >> > > > > > > > > >> > > > running
> >> > > > > > > > > >> > > > >> >> > clients.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
> >> > > generated
> >> > > > > > using
> >> > > > > > > a
> >> > > > > > > > > >> master
> >> > > > > > > > > >> > > key.
> >> > > > > > > > > >> > > > >> The
> >> > > > > > > > > >> > > > >> >> > master
> >> > > > > > > > > >> > > > >> >> > >> >> key
> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be periodically
> >> > > changed.
> >> > > > > In
> >> > > > > > > > > Hadoop,
> >> > > > > > > > > >> the
> >> > > > > > > > > >> > > > >> default
> >> > > > > > > > > >> > > > >> >> > renewal
> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require
> brokers
> >> > > > > > maintaining
> >> > > > > > > a
> >> > > > > > > > > >> list of X
> >> > > > > > > > > >> > > > most
> >> > > > > > > > > >> > > > >> >> > recent
> >> > > > > > > > > >> > > > >> >> > >> >> master
> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to
> be
> >> > > > > persisted
> >> > > > > > > > > >> somewhere, as
> >> > > > > > > > > >> > > > if a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> goes
> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get
> that list
> >> > > > again
> >> > > > > > and
> >> > > > > > > > > >> storing
> >> > > > > > > > > >> > > > master
> >> > > > > > > > > >> > > > >> keys
> >> > > > > > > > > >> > > > >> >> > on ZK
> >> > > > > > > > > >> > > > >> >> > >> >> is
> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However,
> if a
> >> > > > broker
> >> > > > > > goes
> >> > > > > > > > > down
> >> > > > > > > > > >> then
> >> > > > > > > > > >> > > we
> >> > > > > > > > > >> > > > >> have
> >> > > > > > > > > >> > > > >> >> > much
> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and
> client can
> >> > > > always
> >> > > > > > > > > >> > > re-authenticate
> >> > > > > > > > > >> > > > is
> >> > > > > > > > > >> > > > >> such
> >> > > > > > > > > >> > > > >> >> > >> >> events.
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a
> look at
> >> > > other
> >> > > > > > > > > >> alternatives
> >> > > > > > > > > >> > > this
> >> > > > > > > > > >> > > > >> list has
> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough
> proposal,
> >> > > > great
> >> > > > > > > work!"
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at
> 10:28
> >> > PM,
> >> > > > Gwen
> >> > > > > > > > Shapira
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me.
> Thanks!
> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at
> 9:25
> >> > PM,
> >> > > > > > Harsha <
> >> > > > > > > > > >> > > > kafka@harsha.io
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> >> > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any
> release
> >> > > > > vehicle
> >> > > > > > > but
> >> > > > > > > > > >> still the
> >> > > > > > > > > >> > > > >> work can
> >> > > > > > > > > >> > > > >> >> > move
> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> interested in
> >> > the
> >> > > > KIP
> >> > > > > > > > please
> >> > > > > > > > > >> do the
> >> > > > > > > > > >> > > > >> review and
> >> > > > > > > > > >> > > > >> >> > >> >> provide
> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at
> >> > 04:59
> >> > > > PM,
> >> > > > > > > Ismael
> >> > > > > > > > > >> Juma
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would
> be
> >> > good
> >> > > > to
> >> > > > > > have
> >> > > > > > > > > more
> >> > > > > > > > > >> time
> >> > > > > > > > > >> > > to
> >> > > > > > > > > >> > > > >> review
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at
> >> > 12:55
> >> > > > AM,
> >> > > > > > Gwen
> >> > > > > > > > > >> Shapira <
> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48
> depends on
> >> > > > KIP-43,
> >> > > > > > > which
> >> > > > > > > > > is
> >> > > > > > > > > >> > > > already a
> >> > > > > > > > > >> > > > >> bit
> >> > > > > > > > > >> > > > >> >> > of a
> >> > > > > > > > > >> > > > >> >> > >> >> risk
> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release -
> any
> >> > > chance
> >> > > > > we
> >> > > > > > > can
> >> > > > > > > > > >> delay
> >> > > > > > > > > >> > > > >> delegation
> >> > > > > > > > > >> > > > >> >> > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community
> >> > working
> >> > > > on a
> >> > > > > > > > release
> >> > > > > > > > > >> every
> >> > > > > > > > > >> > > 3
> >> > > > > > > > > >> > > > >> month,
> >> > > > > > > > > >> > > > >> >> > this
> >> > > > > > > > > >> > > > >> >> > >> >> is not
> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26,
> 2016 at
> >> > > 5:11
> >> > > > > PM,
> >> > > > > > > > Ashish
> >> > > > > > > > > >> Singh
> >> > > > > > > > > >> > > <
> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for
> the
> >> > > > awesome
> >> > > > > > > write
> >> > > > > > > > > up.
> >> > > > > > > > > >> > > > Following
> >> > > > > > > > > >> > > > >> our
> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it
> will
> >> > be
> >> > > > > easier
> >> > > > > > > to
> >> > > > > > > > > >> compare
> >> > > > > > > > > >> > > > >> various
> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together.
> I am
> >> > > > stating
> >> > > > > > > > below a
> >> > > > > > > > > >> few
> >> > > > > > > > > >> > > > >> >> > alternatives along
> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal)
> >> > Store
> >> > > > > > > Delegation
> >> > > > > > > > > >> Token,
> >> > > > > > > > > >> > > DT,
> >> > > > > > > > > >> > > > >> on ZK.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> > > authenticates
> >> > > > > > with a
> >> > > > > > > > > >> broker.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> client is
> >> > > > > > > > authenticated,
> >> > > > > > > > > >> it
> >> > > > > > > > > >> > > will
> >> > > > > > > > > >> > > > >> make a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> delegation
> >> > > > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> >> > > generates
> >> > > > a
> >> > > > > > > shared
> >> > > > > > > > > >> secret
> >> > > > > > > > > >> > > > based
> >> > > > > > > > > >> > > > >> on
> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret
> >> > shared
> >> > > > > > between
> >> > > > > > > > all
> >> > > > > > > > > >> > > brokers,
> >> > > > > > > > > >> > > > >> >> > randomly
> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker
> stores
> >> > this
> >> > > > > token
> >> > > > > > in
> >> > > > > > > > its
> >> > > > > > > > > >> in
> >> > > > > > > > > >> > > > memory
> >> > > > > > > > > >> > > > >> cache.
> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> DelegationToken
> >> > > > > without
> >> > > > > > > the
> >> > > > > > > > > >> hmac in
> >> > > > > > > > > >> > > the
> >> > > > > > > > > >> > > > >> >> > zookeeper.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers
> will
> >> > > > have a
> >> > > > > > > cache
> >> > > > > > > > > >> backed
> >> > > > > > > > > >> > > by
> >> > > > > > > > > >> > > > >> >> > zookeeper so
> >> > > > > > > > > >> > > > >> >> > >> >> they
> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
> >> > whenever
> >> > > a
> >> > > > > new
> >> > > > > > > > token
> >> > > > > > > > > is
> >> > > > > > > > > >> > > > >> generated and
> >> > > > > > > > > >> > > > >> >> > they
> >> > > > > > > > > >> > > > >> >> > >> >> will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache
> whenever
> >> > > > token
> >> > > > > > > state
> >> > > > > > > > > >> changes.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker
> returns
> >> > the
> >> > > > > token
> >> > > > > > to
> >> > > > > > > > > >> Client.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> and
> >> > fixes
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable
> race
> >> > > > > condition,
> >> > > > > > > > client
> >> > > > > > > > > >> tries
> >> > > > > > > > > >> > > to
> >> > > > > > > > > >> > > > >> >> > authenticate
> >> > > > > > > > > >> > > > >> >> > >> >> with
> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to
> be
> >> > > > updated
> >> > > > > > with
> >> > > > > > > > the
> >> > > > > > > > > >> newly
> >> > > > > > > > > >> > > > >> generated
> >> > > > > > > > > >> > > > >> >> > DT.
> >> > > > > > > > > >> > > > >> >> > >> >> This
> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with
> making
> >> > > > > dtRequest
> >> > > > > > > > block
> >> > > > > > > > > >> until
> >> > > > > > > > > >> > > all
> >> > > > > > > > > >> > > > >> >> > brokers have
> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk
> barrier or
> >> > > > > similar
> >> > > > > > > > > >> mechanism
> >> > > > > > > > > >> > > can
> >> > > > > > > > > >> > > > be
> >> > > > > > > > > >> > > > >> used.
> >> > > > > > > > > >> > > > >> >> > >> >> However,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will
> >> > > increase
> >> > > > > > > > > complexity.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a
> static
> >> > > secret
> >> > > > > key
> >> > > > > > > > from
> >> > > > > > > > > >> config
> >> > > > > > > > > >> > > > >> file. Will
> >> > > > > > > > > >> > > > >> >> > >> >> require
> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and
> uses a
> >> > > static
> >> > > > > > > secret
> >> > > > > > > > > >> key. It
> >> > > > > > > > > >> > > is
> >> > > > > > > > > >> > > > >> advised
> >> > > > > > > > > >> > > > >> >> > to
> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically.
> This
> >> > can
> >> > > > be
> >> > > > > > > > avoided
> >> > > > > > > > > >> with
> >> > > > > > > > > >> > > > >> controller
> >> > > > > > > > > >> > > > >> >> > >> >> generating
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to
> brokers
> >> > > > > > > periodically.
> >> > > > > > > > > >> However,
> >> > > > > > > > > >> > > > >> this will
> >> > > > > > > > > >> > > > >> >> > >> >> require
> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain
> certain
> >> > > counts
> >> > > > of
> >> > > > > > > > > >> secretKeys.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1)
> Have
> >> > > > > controller
> >> > > > > > > > > >> generate
> >> > > > > > > > > >> > > > >> delegation
> >> > > > > > > > > >> > > > >> >> > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> > > authenticates
> >> > > > > > with a
> >> > > > > > > > > >> broker.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> client is
> >> > > > > > > > authenticated,
> >> > > > > > > > > >> it
> >> > > > > > > > > >> > > will
> >> > > > > > > > > >> > > > >> make a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> delegation
> >> > > > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker
> forwards
> >> > the
> >> > > > > > request
> >> > > > > > > > to
> >> > > > > > > > > >> > > > controller.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
> >> > > generates
> >> > > > a
> >> > > > > DT
> >> > > > > > > and
> >> > > > > > > > > >> > > broadcasts
> >> > > > > > > > > >> > > > >> to all
> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker
> stores
> >> > this
> >> > > > > token
> >> > > > > > in
> >> > > > > > > > its
> >> > > > > > > > > >> memory
> >> > > > > > > > > >> > > > >> cache.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller
> >> > responds
> >> > > > to
> >> > > > > > > > broker’s
> >> > > > > > > > > >> DT
> >> > > > > > > > > >> > > req.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker
> returns
> >> > the
> >> > > > > token
> >> > > > > > to
> >> > > > > > > > > >> Client.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> and
> >> > fixes
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will
> have to
> >> > add
> >> > > > new
> >> > > > > > > APIs
> >> > > > > > > > to
> >> > > > > > > > > >> > > support
> >> > > > > > > > > >> > > > >> >> > controller
> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top
> of
> >> > the
> >> > > > > > minimal
> >> > > > > > > > APIs
> >> > > > > > > > > >> that
> >> > > > > > > > > >> > > are
> >> > > > > > > > > >> > > > >> >> > currently
> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will
> also have
> >> > > to
> >> > > > > add
> >> > > > > > > APIs
> >> > > > > > > > > to
> >> > > > > > > > > >> > > support
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new
> broker
> >> > > comes
> >> > > > up
> >> > > > > > it
> >> > > > > > > > will
> >> > > > > > > > > >> have
> >> > > > > > > > > >> > > to
> >> > > > > > > > > >> > > > >> get all
> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
> catastrophic
> >> > > > > failures
> >> > > > > > > > where
> >> > > > > > > > > >> all
> >> > > > > > > > > >> > > > brokers
> >> > > > > > > > > >> > > > >> go
> >> > > > > > > > > >> > > > >> >> > down,
> >> > > > > > > > > >> > > > >> >> > >> >> the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> >> > > servers
> >> > > > > are
> >> > > > > > > > > >> restarted as
> >> > > > > > > > > >> > > > >> tokens
> >> > > > > > > > > >> > > > >> >> > are not
> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> happens,
> >> > then
> >> > > > > there
> >> > > > > > > are
> >> > > > > > > > > more
> >> > > > > > > > > >> > > > important
> >> > > > > > > > > >> > > > >> >> > things to
> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
> better
> >> > to
> >> > > > > > > > > >> re-authenticate.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2)
> Do not
> >> > > > > > distribute
> >> > > > > > > > DT
> >> > > > > > > > > to
> >> > > > > > > > > >> > > other
> >> > > > > > > > > >> > > > >> brokers
> >> > > > > > > > > >> > > > >> >> > at
> >> > > > > > > > > >> > > > >> >> > >> >> all.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> > > authenticates
> >> > > > > > with a
> >> > > > > > > > > >> broker.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> client is
> >> > > > > > > > authenticated,
> >> > > > > > > > > >> it
> >> > > > > > > > > >> > > will
> >> > > > > > > > > >> > > > >> make a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> delegation
> >> > > > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> >> > > generates
> >> > > > DT
> >> > > > > > of
> >> > > > > > > > > form,
> >> > > > > > > > > >> > > [hmac +
> >> > > > > > > > > >> > > > >> (owner,
> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime,
> id,
> >> > hmac,
> >> > > > > > > > > >> expirationTime)]
> >> > > > > > > > > >> > > and
> >> > > > > > > > > >> > > > >> passes
> >> > > > > > > > > >> > > > >> >> > back
> >> > > > > > > > > >> > > > >> >> > >> >> this
> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is
> generated
> >> > via
> >> > > > > > > > > >> {HMAC-SHA256(owner,
> >> > > > > > > > > >> > > > >> renewer,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime)
> >> > using
> >> > > > > > > > SecretKey}.
> >> > > > > > > > > >> Note
> >> > > > > > > > > >> > > that
> >> > > > > > > > > >> > > > >> all
> >> > > > > > > > > >> > > > >> >> > brokers
> >> > > > > > > > > >> > > > >> >> > >> >> have
> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then
> goes
> >> > to
> >> > > > any
> >> > > > > > > > broker
> >> > > > > > > > > >> and to
> >> > > > > > > > > >> > > > >> >> > authenticate
> >> > > > > > > > > >> > > > >> >> > >> >> sends
> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
> recalculates
> >> > > hmac
> >> > > > > > using
> >> > > > > > > > > >> (owner,
> >> > > > > > > > > >> > > > >> renewer,
> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> expirationTime) info
> >> > > > from
> >> > > > > DT
> >> > > > > > > and
> >> > > > > > > > > its
> >> > > > > > > > > >> > > > >> SecretKey. If
> >> > > > > > > > > >> > > > >> >> > it
> >> > > > > > > > > >> > > > >> >> > >> >> matches
> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> >> > > > > authenticated.
> >> > > > > > > > Yes,
> >> > > > > > > > > >> it will
> >> > > > > > > > > >> > > > do
> >> > > > > > > > > >> > > > >> other
> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp
> expiry and
> >> > > > such.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret
> key
> >> > will
> >> > > > be
> >> > > > > > > > > generated
> >> > > > > > > > > >> by
> >> > > > > > > > > >> > > > >> controller
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> passed
> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> and
> >> > fixes
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to
> delete a
> >> > DT?
> >> > > > > Yes,
> >> > > > > > > that
> >> > > > > > > > > is
> >> > > > > > > > > >> a
> >> > > > > > > > > >> > > > downside
> >> > > > > > > > > >> > > > >> >> > here.
> >> > > > > > > > > >> > > > >> >> > >> >> However,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with
> >> > > brokers
> >> > > > > > > > > maintaining
> >> > > > > > > > > >> a
> >> > > > > > > > > >> > > > >> blacklist of
> >> > > > > > > > > >> > > > >> >> > DTs,
> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed
> after
> >> > > > > expiry.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
> catastrophic
> >> > > > > failures
> >> > > > > > > > where
> >> > > > > > > > > >> all
> >> > > > > > > > > >> > > > brokers
> >> > > > > > > > > >> > > > >> go
> >> > > > > > > > > >> > > > >> >> > down,
> >> > > > > > > > > >> > > > >> >> > >> >> the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> >> > > servers
> >> > > > > are
> >> > > > > > > > > >> restarted as
> >> > > > > > > > > >> > > > >> tokens
> >> > > > > > > > > >> > > > >> >> > are not
> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> happens,
> >> > then
> >> > > > > there
> >> > > > > > > are
> >> > > > > > > > > more
> >> > > > > > > > > >> > > > important
> >> > > > > > > > > >> > > > >> >> > things to
> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
> better
> >> > to
> >> > > > > > > > > >> re-authenticate.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26,
> 2016 at
> >> > > > 1:58
> >> > > > > > PM,
> >> > > > > > > > > Parth
> >> > > > > > > > > >> > > > >> Brahmbhatt <
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > pbrahmbhatt@hortonworks.com>
> >> > > > > > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed
> KIP-48 so
> >> > > we
> >> > > > > can
> >> > > > > > > > offer
> >> > > > > > > > > >> hadoop
> >> > > > > > > > > >> > > > like
> >> > > > > > > > > >> > > > >> >> > delegation
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can
> review
> >> > > the
> >> > > > > > design
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends
> on
> >> > > KIP-43
> >> > > > > and
> >> > > > > > > we
> >> > > > > > > > > >> have also
> >> > > > > > > > > >> > > > >> >> > discussed an
> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design
> here<
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Regards,
> >> > > > > >
> >> > > > > > Rajini
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Regards,
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Liquan Pei
> >> Software Engineer, Confluent Inc
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message