kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Becket Qin <becket....@gmail.com>
Subject Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API
Date Tue, 17 Oct 2017 03:50:10 GMT
Hi Paolo,

Thanks for the KIP and sorry for being late on the thread. I am wondering
what is the KafkaFuture<Long> returned by all() call? Should it be a
Map<TopicPartition, Long> instead?

Thanks,

Jiangjie (Becket) QIn

On Thu, Sep 28, 2017 at 3:48 AM, Paolo Patierno <ppatierno@live.com> wrote:

> Hi,
>
>
> maybe we want to start without the delete records policy for now waiting
> for a real needs. So I'm removing it from the KIP.
>
> I hope for more comments on this KIP-204 so that we can start a vote on
> Monday.
>
>
> Thanks.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Paolo Patierno <ppatierno@live.com>
> Sent: Thursday, September 28, 2017 5:56 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi,
>
>
> I have just updated the KIP-204 description with the new
> TopicDeletionPolicy suggested by the KIP-201.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Paolo Patierno <ppatierno@live.com>
> Sent: Tuesday, September 26, 2017 4:57 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Tom,
>
> as I said in the KIP-201 discussion I'm ok with having a unique
> DeleteTopicPolicy but then it could be useful having more information then
> just the topic name; partitions and offset for messages deletion could be
> useful for a fine grained use cases.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Tom Bentley <t.j.bentley@gmail.com>
> Sent: Tuesday, September 26, 2017 4:32 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Paolo,
>
> I guess a RecordDeletionPolicy should work at the partition level, whereas
> the TopicDeletionPolicy should work at the topic level. But then we run
> into a similar situation as described in the motivation for KIP-201, where
> the administrator might have to write+configure two policies in order to
> express their intended rules. For example, it's no good preventing people
> from deleting topics if they can delete all the messages in those topics,
> or vice versa.
>
> On that reasoning, perhaps there should be a single policy interface
> covering topic deletion and message deletion. Alternatively, the topic
> deletion API could also invoke the record deletion policy (before the topic
> deletion policy I mean). But the former would be more consistent with
> what's proposed in KIP-201.
>
> Wdyt? I can add this to KIP-201 if you want.
>
> Cheers,
>
> Tom
>
>
>
>
>
> On 26 September 2017 at 17:01, Paolo Patierno <ppatierno@live.com> wrote:
>
> > Hi Tom,
> >
> > I think that we could live with the current authorizer based on delete
> > topic (for both, deleting messages and topic as a whole) but then the
> > RecordsDeletePolicy could be even more fine grained giving the
> possibility
> > to avoid deleting messages for specific partitions (inside the topic) and
> > starting from a specific offset.
> >
> > I could think on some users solutions where they know exactly what the
> > partitions means inside of a specific topic (because they are using a
> > custom partitioner on the producer side) so they know what kind of
> messages
> > are inside a partition allowing to delete them but not the other.
> >
> > In such a policy a user could also check the timestamp related to the
> > offset for allowing or not deletion on time base.
> >
> >
> > Wdyt ?
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > ________________________________
> > From: Tom Bentley <t.j.bentley@gmail.com>
> > Sent: Tuesday, September 26, 2017 2:55 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > new Admin Client API
> >
> > Hi Edoardo and Paolo,
> >
> >
> > On 26 September 2017 at 14:10, Paolo Patierno <ppatierno@live.com>
> wrote:
> >
> > > What could be useful use cases for having a RecordsDeletePolicy ?
> Records
> > > can't be deleted for a topic name ? Starting from a specific offset ?
> > >
> >
> > I can imagine some users wanting to prohibit using this API completely.
> > Maybe others divide up the topic namespace according to some scheme, and
> so
> > it would be allowed for some topics, but not for others based on the
> name.
> > Both these could be done using authz, but would be much simpler to
> express
> > using a policy.
> >
> > Since both deleting messages and deleting topics are authorized using
> > delete operation on the topic, using policies it would be possible to
> allow
> > deleting messages from a topic, but not deleting the topic itself.
> >
> >
> > On 26 September 2017 at 15:27, Edoardo Comar <ECOMAR@uk.ibm.com> wrote:
> >
> > >
> > > Our KIP-170 did indeed suggest a TopicDeletePolicy - but, as said, for
> > our
> > > intent an Authorizer implementation will be usable instead,
> > >
> >
> > I guess authorization in the most general sense encompass es both the
> > ACL-based authorization inherent in Authorizer and the various
> > operation-specific *Policies. But they're not the same. The Policies are
> > about deciding on *what* is requested, and the Authorizer is about
> making a
> > decision purely on *who* is making the request. It's quite legitimate to
> > want to use both, or just one or the other.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message