kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Patierno <ppatie...@live.com>
Subject Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
Date Wed, 27 Sep 2017 10:45:05 GMT
Hi Ismael,


  1.  I don't have a real requirement now but "deleting" is an operation that could be really
dangerous so it's always better having a way for having more control on that. I know that
we have the authorizer used for that (delete on topic) but fine grained control could be better
(even already happens for topic deletion).
  2.  I know about the problem of restarting broker due to changes on policies but what do
you mean by doing that on the clients ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


________________________________
From: ismaelj@gmail.com <ismaelj@gmail.com> on behalf of Ismael Juma <ismael@juma.me.uk>
Sent: Wednesday, September 27, 2017 10:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

A couple of questions:

1. Is this a concrete requirement from a user or is it hypothetical?
2. You sure you would want to do this in the broker instead of the clients?
It's worth remembering that updating broker policies involves a rolling
restart of the cluster, so it's not the right place for things that change
frequently.

Ismael

On Wed, Sep 27, 2017 at 11:26 AM, Paolo Patierno <ppatierno@live.com> wrote:

> Hi Ismael,
>
> regarding motivations for delete records, as I said during the discussion
> on KIP-204, it gives the possibility to avoid deleting messages for
> specific partitions (inside the topic) and starting from a specific offset.
> I could think on some users solutions where they know exactly what the
> partitions means in a specific topic (because they are using a custom
> partitioner on the producer side) so they know what kind of messages are
> inside a partition allowing to delete them but not the others.  In such a
> policy a user could also check the timestamp related to the offset for
> allowing or not deletion on time base.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: ismaelj@gmail.com <ismaelj@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
> Sent: Wednesday, September 27, 2017 10:18 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
>
> A couple more comments:
>
> 1. "If this KIP is accepted for Kafka 1.1.0 this removal could happen in
> Kafka 1.2.0 or a later release." -> we only remove code in major releases.
> So, if it's deprecated in 1.1.0, it would be removed in 2.0.0.
>
> 2. Deleting all messages in a topic is not really the same as deleting a
> topic. The latter will cause consumers and producers to error out while the
> former will not. It would be good to motivate the need for the delete
> records policy more.
>
> Ismael
>
> On Wed, Sep 27, 2017 at 11:12 AM, Ismael Juma <ismael@juma.me.uk> wrote:
>
> > Another quick comment: the KIP states that having multiple interfaces
> > imply that the logic must be in 2 places. That is not true because the
> same
> > class can implement multiple interfaces (this aspect was considered when
> we
> > decided to introduce policies incrementally).
> >
> > The main reason why I think the original approach doesn't work well is
> > that there is no direct mapping between an operation and the policy. That
> > is, we initially thought we would have create/alter/delete topics, but
> that
> > didn't work out as the alter case is better served by multiple request
> > types. Given that, it's a bit awkward to maintain the original approach
> and
> > a policy for topic management seemed easier to understand. On that note,
> > would `TopicManagementPolicy` be a better name?
> >
> > Looking at the updated KIP, I notice that we actually have a
> > TopicDeletionPolicy with a separate config. That seems to be a halfway
> > house. Not sure about that.
> >
> > Ismael
> >
> > On Wed, Sep 27, 2017 at 10:47 AM, Tom Bentley <t.j.bentley@gmail.com>
> > wrote:
> >
> >> I have updated the KIP to add a common policy interface for topic and
> >> message deletion. This included pulling ClusterState and TopicState
> >> interfaces up to the top level so that they can be shared between the
> two
> >> policies.
> >>
> >> Cheers,
> >>
> >> Tom
> >>
> >> On 26 September 2017 at 18:09, Edoardo Comar <ECOMAR@uk.ibm.com> wrote:
> >>
> >> > Thanks Tom,
> >> > In my original KIP-170 I mentioned that the method
> >> >
> >> > public Map<String, Integer> topicsPartitionCount();
> >> >
> >> > was just a starting point for a general purpose ClusterState
> >> > as it happened to be exactly the info we needed for our policy
> >> > implementation :-)
> >> > it definitely doesn't feel general purpose enough.
> >> >
> >> > what about
> >> >
> >> >     interface ClusterState {
> >> >         public TopicState topicState(String topicName);
> >> >         public Set<String> topics();
> >> >     }
> >> >
> >> > I think that the implementation of ClusterState that the server will
> >> pass
> >> > to the policy.validate method
> >> > would just lazily tap into MetadataCache. No need for big upfront
> >> > allocations.
> >> >
> >> > ciao,
> >> > Edo
> >> > --------------------------------------------------
> >> >
> >> > Edoardo Comar
> >> >
> >> > IBM Message Hub
> >> >
> >> > IBM UK Ltd, Hursley Park, SO21 2JN
> >> >
> >> >
> >> >
> >> > From:   Tom Bentley <t.j.bentley@gmail.com>
> >> > To:     dev@kafka.apache.org
> >> > Date:   26/09/2017 17:39
> >> > Subject:        Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
> >> >
> >> >
> >> >
> >> > Hi Edoardo,
> >> >
> >> >
> >> > what about a single method in ClusterState
> >> > >
> >> > >     interface ClusterState {
> >> > >         public Map<String,TopicState> topicsState();
> >> > >
> >> > >     }
> >> > >
> >> > > which could return a read-only snapshot of the cluster metadata ?
> >> > >
> >> >
> >> > Sure that would work too. A concern with that is that we end up
> >> allocating
> >> > a potentially rather large amount for the Map and the collections
> >> present
> >> > in the TopicStates in order to provide the snapshot. The caller might
> >> only
> >> > be interested in one item from the TopicState for one topic in the
> map.
> >> > Accessing this information via methods means the caller only pays for
> >> what
> >> > they use.
> >> >
> >> > Cheers,
> >> >
> >> > Tom
> >> >
> >> >
> >> >
> >> > Unless stated otherwise above:
> >> > IBM United Kingdom Limited - Registered in England and Wales with
> number
> >> > 741598.
> >> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> >> 3AU
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message