kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: [DISCUSS] KIP-37 - Add namespaces in Kafka
Date Tue, 13 Oct 2015 02:37:50 GMT
This works really nicely from the consumer side, but what about the
producer? If there are no more topics,do we allow producing to a directory
and have the Partitioner hash-partition messages between all partitions in
the multiple levels in a directory?

Also, I think we want to preserve the consumer terminology of "subscribe"
to topics / directories, but "assign" partitions - since the consumer
behavior is different in those cases.

On Mon, Oct 12, 2015 at 7:16 PM, Jay Kreps <jay@confluent.io> wrote:

> Okay this is similar to what I think we have talked about before. Let me
> elaborate on the idea that I think has been floating around--it's pretty
> similar with a few differences.
>
> I think what you are calling the "default namespace" is basically what I
> would call the "current working directory" with paths not beginning with
> '/' being interpreted relative to this directory as in the fs.
>
> One thing you have to work out is what levels in this hierarchy you can
> actually subscribe to. I think you are assuming only what we currently
> consider a "topic", i.e. the first level of directories but not the
> partitions or parent dirs, would be subscribable. If you think about it,
> though, that constraint is a bit arbitrary.
>
> I'd propose instead the semantics that:
> - Subscribing to /a/b/c/0 means subscribing to the 0th partition of topic
> "c" in directory /a/b
> - Subscribing to /a/b/c means subscribing to all partitions in
> topic/directory "c"
> - Subscribing to /a/b means subscribing to all partitions in all
> topics/subdirectories under a/b recursively
>
> Effectively the concept of topics goes away entirely--you just have
> partitions/logs and directories. In this respect rather than adding new
> concepts this new feature would actually just generalizes what we have
> (which I think is a good thing).
>
> -Jay
>
> On Mon, Oct 12, 2015 at 6:24 PM, Ashish Singh <asingh@cloudera.com> wrote:
>
> > On Mon, Oct 12, 2015 at 5:42 PM, Jay Kreps <jay@confluent.io> wrote:
> >
> > > Great. I definitely would strongly favor carrying over user's intuition
> > > from FS unless we think we need a very different model. The minor
> details
> > > like the seperator and namespace term will help with that.
> > >
> > > Follow-up question, say I have a layout like
> > >    /chicago-datacenter/user-events/pageviews
> > > Can I subscribe to
> > >    /chicago-datacenter/user-events
> > >
> > Yes, however they will have need a regex like
> > /chicago-datacenter/user-events/*
> >
> > > to get the full firehose of user events from chicago? Can I subscribe
> to
> > >    /*/user-events
> > > to get user events originating from all datacenters?
> > >
> > Yes, however they will have need a regex like
> > /chicago-datacenter/user-events/*
> > Yes
> >
> > >
> > > (Assuming, for now, that these are all in the same cluster...)
> > >
> > > Also, just to confirm, it sounds from the proposal like config
> overrides
> > > would become fully hierarchical so you can override config at any
> > directory
> > > point. This will add complexity in implementation but I think will
> likely
> > > be much more operator friendly.
> > >
> > Yes, that is the idea.
> >
> > >
> > > There are about a thousand details to discuss in terms of how this
> would
> > > impact the metadata request, various zk entries, and various other
> > aspects,
> > > but probably it makes sense to first agree on how we would want it to
> > work
> > > and then start to dive into how to implement that.
> > >
> > Agreed.
> >
> > >
> > > -Jay
> > >
> > > On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh <asingh@cloudera.com>
> > wrote:
> > >
> > > > Hey Jay, thanks for reviewing the proposal. Answers inline.
> > > >
> > > > On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps <jay@confluent.io>
> wrote:
> > > >
> > > > > Hey guys,
> > > > >
> > > > > I think this is an important feature and one we've talked about
> for a
> > > > > while. I really think trying to invent a new nomenclature is going
> to
> > > > make
> > > > > it hard for people to understand, though. As such I recommend we
> call
> > > > > namespaces "directories" and denote them with '/'--this will make
> the
> > > > > feature 1000x more understandable to people.
> > > >
> > > > Essentially you are suggesting two things here.
> > > > 1. Use "Directory" instead of "Namespace" as it is more intuitive. I
> > > agree.
> > > > 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if
> we
> > > > call these directories, '/' is the way to go.
> > > >
> > > > I think we should inheret the
> > > > > semantics of normal unix fs in so far as it makes sense.
> > > > >
> > > > > In this approach we get rid of topics entirely, instead we really
> > just
> > > > have
> > > > > partitions which are the equivalent of a file and retain their
> > numeric
> > > > > names, and the existing topic concept is just the first directory
> > level
> > > > but
> > > > > we generalize to allow arbitrarily many more levels of nesting.
> This
> > > > allows
> > > > > categorization of data, such as
> /datacenter1/user-events/page-views/3
> > > and
> > > > > you can subscribe, apply configs or permissions at any level of the
> > > > > hierarchy.
> > > > >
> > > > +1. This actually requires just a minor change to existing proposal,
> > > i.e.,
> > > > "some:namespace:topic" becomes "some/namespace/topic".
> > > >
> > > > >
> > > > > I'm actually not 100% such what the semantics of accessing data in
> > > > > differing namespaces is in the current proposal, maybe you can
> > clarify
> > > > > Ashish?
> > > >
> > > > I will add more info to KIP on this, however I think a client should
> be
> > > > able to access data in any namespace as long as following conditions
> > are
> > > > satisfied.
> > > >
> > > > 1. Namespace, the client is trying to access, exists.
> > > > 2. The client has sufficient permissions on the namespace for type of
> > > > operation the client is trying to perform on a topic within that
> > > namespace.
> > > > 3. The client has sufficient permissions on the topic for type of
> > > operation
> > > > the client is trying to perform on that topic.
> > > >
> > > > If we choose to go with what you suggested earlier that just have
> > > hierarchy
> > > > of directories, then step 3 will actually be covered in step 2.
> > > >
> > > > In the current proposal, consumers will subscribe to a topic in a
> > > namespace
> > > > by specifying <namespace>:<topic> as the topic name. They
can
> subscribe
> > > to
> > > > topics from multiple namespaces.
> > > >
> > > > Let me know if I totally missed your question.
> > > >
> > > > Since the point of Kafka is sharing data I think it is really
> > > > > important that the grouping be just for
> > > > convenience/permissions/config/etc
> > > > > and that it remain possible to access multiple
> directories/namespaces
> > > > from
> > > > > the same client.
> > > > >
> > > > Totally agree with you.
> > > >
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh <asingh@cloudera.com>
> > > > wrote:
> > > > >
> > > > > > Hey Guys,
> > > > > >
> > > > > > I just created KIP-37 for adding namespaces to Kafka.
> > > > > >
> > > > > > KIP-37
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > > > > > >
> > > > > > tracks the proposal.
> > > > > >
> > > > > > The idea is to make Kafka support multi-tenancy via namespaces.
> > > > > >
> > > > > > Feedback and comments are welcome.
> > > > > > ‚Äč
> > > > > > --
> > > > > >
> > > > > > Regards,
> > > > > > Ashish
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message