Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOJcB38774Etw8iYDjpyqyQqay8vSv-PBDHfbHgidFJkeNM5JA@mail.gmail.com>
References: <CAOJcB3_rvsyebgm1bP9a9CFBpWEcGdpL9_1GH4O+_ym+EED7Kw@mail.gmail.com>
	<CAOeJiJiYh+VnEBavugk1zoBW2xy5f-Ym=yv1nJJ-m3T4=uC=xQ@mail.gmail.com>
	<CAOJcB38HFoumudqFFu2mb+NnWq1wjrz4mQT0wiSxRSB7z-sQVg@mail.gmail.com>
	<CAOeJiJh8t2gGCoCe4tjRv28ADAZ1i9eaaxvmHMKS2Kx6=CmB4A@mail.gmail.com>
	<CAA9QSJKVnpR-qZWiYyVOEvTLYnHb=b6waRcg0OyLVJa63M3+CA@mail.gmail.com>
	<CAOJcB3-kR1Ae9qr7t4uWBCM_90NiVCH1OrdW_S4EoA4Jhi9AxQ@mail.gmail.com>
	<CAOeJiJgubhk5JCSVY2p6L+PuLJffiHCjsDMZvgLXDiMFNVhZLw@mail.gmail.com>
	<CAOJcB38774Etw8iYDjpyqyQqay8vSv-PBDHfbHgidFJkeNM5JA@mail.gmail.com>
Date: Fri, 10 Jun 2016 15:28:11 -0700
Message-ID: <CAFc58G8E-2Yt5dL6EqET5uzROGzhYjXBB6dzrOT7zGB2A+ttyg@mail.gmail.com>
Subject: Re: [VOTE] KIP-55: Secure quotas for authenticated users
From: Jun Rao <jun@confluent.io>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Content-Type: multipart/alternative; boundary=001a11445584cf2c6c0534f40d48
archived-at: Fri, 10 Jun 2016 22:28:29 -0000

--001a11445584cf2c6c0534f40d48
Content-Type: text/plain; charset=UTF-8

Rajini,

The new proposal sounds good to me too. My only question is what happens to
those existing quotas on client-id. Do we just treat them as the quota for
that client-id under ANONYMOUS user name?

Thanks,

Jun

On Fri, Jun 10, 2016 at 2:43 PM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:

> Jay,
>
> Thank you for the quick feedback. It shouldn't be too hard since I had a PR
> earlier along these lines anyway.
>
> Jun, are you ok with this approach? If everyone agrees, I will close this
> vote, update the KIP and give some more time for discussions.
>
>
> On Fri, Jun 10, 2016 at 10:27 PM, Jay Kreps <jay@confluent.io> wrote:
>
> > This sounds a lot better to me--hopefully it isn't too much harder! I do
> > think if it is possible to do this directly that will be better for users
> > than having an intermediate step since we always have to work through
> > migrating people who have setup quotas already from the old way to the
> new
> > way.
> >
> > -Jay
> >
> > On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram <
> > rajinisivaram@googlemail.com> wrote:
> >
> > > I do think client-id is a valid and useful grouping for quotas even in
> > > secure clusters - but only if clientA of user1 is treated as a
> different
> > > client-id from clientA of user2. Grouping of clients of a user enables
> > > users to allocate their quota effectively to their clients (eg.
> guarantee
> > > that critical event processing clients are not throttled by auditing
> > > clients). When the KIP was down-sized to support only user-based
> quotas,
> > I
> > > was hoping that we could extend it at a later time to enable
> hierarchical
> > > quotas. But I understand that it can be confusing to switch the
> semantics
> > > of quotas based on modes set in the brokers. So let me try once again
> to
> > > promote the original KIP-55. At the time, I did have a flag to revert
> to
> > > the existing client-id behavior to maintain compatibility. But perhaps
> > that
> > > is not necessary.
> > >
> > > How does this sound?
> > >
> > >    - Quotas may be configured for users. Sub-quotas may be configured
> for
> > >    client-ids of a user. Quotas may also be configured for client-ids
> of
> > > users
> > >    with unlimited quota (Long.MaxValue).
> > >    - Users who don't have a quota override are allocated a configurable
> > >    default quota.
> > >    - Client-ids without a sub-quota override share the remainder of the
> > >    user quota if the user has a quota limit. Default quotas may be
> > defined
> > > for
> > >    clients of users with unlimited quota.
> > >    - For an insecure or single-user secure cluster, the existing
> > client-id
> > >    based quota semantics can be achieved by configuring unlimited quota
> > for
> > >    the user and sub-quota configuration for client-id that matches the
> > > current
> > >    client-id quota configuration.
> > >    - For a cluster mixes both secure and insecure access, client-id
> > quotas
> > >    can be set for unauthenticated clients (unlimited quota for
> ANONYMOUS,
> > >    quotas for client-ids) and user quotas can be set for authenticated
> > > users.
> > >    - In a multi-user cluster, it is currently possible to define quotas
> > for
> > >    client-ids that span multiple users. This will no longer be
> supported.
> > >
> > >
> > >
> > >
> > > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira <gwen@confluent.io>
> wrote:
> > >
> > > > I am not crazy about modes either. An earlier proposal supported both
> > > > client-ids and users at the same time, and it made more sense to me.
> I
> > > > believe it was dropped without proper discussion (Basically, Jun
> > > > mentioned it is complex and Rajini agreed to drop it). We should
> > > > probably rethink the complexity of supporting both vs the limitations
> > > > of "modes".
> > > >
> > > > As you said, we will have secure clients authenticating with users
> and
> > > > insecure clients authenticating with client-ids at the same time.
> > > >
> > > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps <jay@confluent.io> wrote:
> > > > > Hey Rajini,
> > > > >
> > > > > 1. That makes sense to me. Is that reflected in the documentation
> > > > anywhere
> > > > > (I couldn't really find it)? Is there a way to discover that
> > > definition?
> > > > We
> > > > > do way better when we right this stuff down so it has an official
> > > > > definition users and developers can work off of...
> > > > > 2. If client id is a thing that makes sense even when you have
> users,
> > > why
> > > > > would you not want to quota on it?
> > > > >
> > > > > I am not wild about these "modes" where you boot the cluster in
> mode
> > X
> > > > and
> > > > > it behaves in one way and in mode Y and it behaves in another. It
> is
> > > > > complex then for users who expect to be able to set quotas but then
> > > have
> > > > to
> > > > > be able to get access to the filesystem of the kafka nodes to
> > discover
> > > > what
> > > > > mode the cluster is in to know which kind of quota is applicable.
> > > > >
> > > > > I guess there are two ways to think about a feature: one is the
> > > increment
> > > > > from where we are, and the other is the resulting state after that
> > > > > increment is taken. What I am asking is not "is this a low cost
> step
> > > from
> > > > > where we are?" which everyone can agree that it is, but rather
> "does
> > > this
> > > > > make sense as an end state--i.e. if you were starting fresh with
> > > neither
> > > > > users nor client ids nor quotas would you end up with this?".
> > > > >
> > > > > In terms of use cases, I think that we support mixing secure and
> > > insecure
> > > > > access on a single cluster so presumably in that case you would
> want
> > to
> > > > be
> > > > > able to quota insecure users based on client id and secure users
> > based
> > > on
> > > > > user, right? Likewise, as you said, client id is a valid grouping
> > even
> > > in
> > > > > the presence of users, so it might be the case that several apps
> that
> > > are
> > > > > all part of the same system might access Kafka under a single user,
> > but
> > > > you
> > > > > might have different quotas for these different apps. Basically if
> > > client
> > > > > id is a valid grouping even in the presence of users (willing to
> > debate
> > > > > this point, btw!) then you would want to quota on it.
> > > > >
> > > > > -Jay
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com> wrote:
> > > > >
> > > > >> Jay,
> > > > >>
> > > > >> Thank you for the feedback.
> > > > >>
> > > > >> 1. I think it is good to have a single concept of identity, but
> > > multiple
> > > > >> ways of grouping clients for different functions. Client-id is a
> > > logical
> > > > >> grouping of clients with a meaningful name that is used in client
> > > > metrics
> > > > >> and logs. User principal is an authenticated user or a grouping of
> > > > >> unauthenticated users chosen by the broker and is used for ACLs.
> In
> > my
> > > > >> view, in a multi-user system, there is a hierarchy - user owns
> zero
> > or
> > > > more
> > > > >> clients. (principal, client-id) defines a safe group, but the
> > shorter
> > > > >> unsafe client-id is sufficient in client metrics and logs.
> > > > >>
> > > > >>
> > > > >> 2. KIP-55 was initially written to support hierarchical quotas
> > (quotas
> > > > for
> > > > >> user1-clientA, user2-clientA etc), but Jun's view was that the
> > > > complexity
> > > > >> was not justified since there is no clear requirement for this.
> The
> > > > >> cut-down version of the KIP is clearly a lot simpler. But I think
> > your
> > > > >> suggestion is to enable non-hierarchical user quotas and client-id
> > > > quotas
> > > > >> at the same time. Basically treat users and client-ids as distinct
> > > > entities
> > > > >> like topics and allow quotas to be applied to each of these
> > entities.
> > > I
> > > > >> agree that we want to support quotas simultaneously on different
> > > > entities
> > > > >> like topics and users. I am less convinced of client-id and user
> > being
> > > > >>
> > > > >> distinct entities that require separate quotas at the same time.
> And
> > > > >> treating client-id and user as distinct entities makes it harder
> to
> > > > >> implement hierarchical quotas in future.
> > > > >>
> > > > >>
> > > > >> A single user system needs only client-id quotas, and multi-tenant
> > > > system
> > > > >> cannot use client-id quotas since we need to guarantee that one
> > > tenant's
> > > > >> quotas can never be used by another tenant. I suppose a multi-user
> > > > cluster
> > > > >> where users trust each other could apply separate quotas for both
> > > > clients
> > > > >> and users, but I am not sure if there is a usecase that can't be
> > > > satisfied
> > > > >> with just client-id based quotas for this case. Do you have a
> > usecase
> > > in
> > > > >> mind where you want to apply separate quotas for clients and users
> > > > >> simultaneously?
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps <jay@confluent.io>
> wrote:
> > > > >>
> > > > >> > Super sorry to come in late on this one. Rajini, I had two quick
> > > > >> questions
> > > > >> > I think we should be able to answer:
> > > > >> >
> > > > >> >    1. Do client ids make sense in a world which has users? If
> not
> > > > should
> > > > >> we
> > > > >> >    unify them the way Hadoop did (without auth the user is a
> kind
> > of
> > > > best
> > > > >> >    effort honor system identity). This came up in the discussion
> > > > thread
> > > > >> > but I
> > > > >> >    didn't really see a crisp answer. Basically, what is the
> > > > definition of
> > > > >> >    "client id" and what is the definition of "user" and how do
> the
> > > > >> concepts
> > > > >> >    relate?
> > > > >> >    2. If both client ids and users are sensible distinct notions
> > and
> > > > we
> > > > >> >    want to maintain both, why don't we just support quotas on
> > both?
> > > If
> > > > >> they
> > > > >> >    both make sense then you would have a reason to set quotas at
> > > both
> > > > >> > levels.
> > > > >> >    Why have this "mode" that you set that swaps between only
> being
> > > > able
> > > > >> to
> > > > >> > use
> > > > >> >    one and the other? I should be able to set quotas at both
> > levels.
> > > > >> Going
> > > > >> >    forward the model we had discussed with quotas was
> potentially
> > > > being
> > > > >> > able
> > > > >> >    to set quotas for many things independently (say at the topic
> > > > level),
> > > > >> > and I
> > > > >> >    don't think it would make sense to extend this mode approach
> to
> > > > those.
> > > > >> >
> > > > >> > -Jay
> > > > >> >
> > > > >> > On Wed, Jun 8, 2016 at 12:56 PM, Rajini Sivaram <
> > > > >> > rajinisivaram@googlemail.com> wrote:
> > > > >> >
> > > > >> > > I would like to initiate the vote for KIP-55.
> > > > >> > >
> > > > >> > > The KIP details are here: KIP-55: Secure quotas for
> > authenticated
> > > > users
> > > > >> > > <
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> > > > >> > > >
> > > > >> > > .
> > > > >> > >
> > > > >> > > The JIRA  KAFKA-3492  <
> > > > >> https://issues.apache.org/jira/browse/KAFKA-3492
> > > > >> > > >has
> > > > >> > > a draft PR here: https://github.com/apache/kafka/pull/1256.
> > > > >> > >
> > > > >> > > Thank you...
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > >
> > > > >> > > Rajini
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >>
> > > > >> Rajini
> > > > >>
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

--001a11445584cf2c6c0534f40d48--