kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <...@confluent.io>
Subject Re: [DISCUSS] KIP-37 - Add namespaces in Kafka
Date Mon, 12 Oct 2015 17:53:17 GMT
Hey guys,

I think this is an important feature and one we've talked about for a
while. I really think trying to invent a new nomenclature is going to make
it hard for people to understand, though. As such I recommend we call
namespaces "directories" and denote them with '/'--this will make the
feature 1000x more understandable to people. I think we should inheret the
semantics of normal unix fs in so far as it makes sense.

In this approach we get rid of topics entirely, instead we really just have
partitions which are the equivalent of a file and retain their numeric
names, and the existing topic concept is just the first directory level but
we generalize to allow arbitrarily many more levels of nesting. This allows
categorization of data, such as /datacenter1/user-events/page-views/3 and
you can subscribe, apply configs or permissions at any level of the
hierarchy.

I'm actually not 100% such what the semantics of accessing data in
differing namespaces is in the current proposal, maybe you can clarify
Ashish? Since the point of Kafka is sharing data I think it is really
important that the grouping be just for convenience/permissions/config/etc
and that it remain possible to access multiple directories/namespaces from
the same client.

-Jay

On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh <asingh@cloudera.com> wrote:

> Hey Guys,
>
> I just created KIP-37 for adding namespaces to Kafka.
>
> KIP-37
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> >
> tracks the proposal.
>
> The idea is to make Kafka support multi-tenancy via namespaces.
>
> Feedback and comments are welcome.
> ‚Äč
> --
>
> Regards,
> Ashish
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message