kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: auto-delete empty, old topics?
Date Thu, 14 Mar 2013 21:17:04 GMT
Also,

I see a bazillion consecutive log lines like this:

2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
Clearing the current data chunk for this consumer iterator

With the same content (not sure how useful that is!).

Jason

On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <jbr@squareup.com> wrote:

> Hi Neha,
>
> So I did this, but I still see the full list of topics (most of which have
> been deleted), in the consumer logs, e.g.:
>
> consumer.ZookeeperConsumerConnector -
> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> consume = List(<a bazillion topics, most of which have been deleted listed
> here>)
>
> I select topics using the white list topic selector (using a wild-card
> pattern, etc.).
>
> Thoughts?
>
> Does zookeeper keep separate info for the consumer, that needs to be
> removed?
>
> Jason
>
> On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:
>
>> In Kafka 0.7.2, data log directory delete and then rolling restart will
>> work to get rid of the topic completely from zookeeper as well as the
>> broker.
>> This is because the broker registers the topics and partitions from its
>> data directory into zookeeper on startup.
>>
>> Thanks,
>> Neha
>>
>>
>> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <jbr@squareup.com> wrote:
>>
>> > Thanks Neha,
>> >
>> > So are you saying that on 0.7.2, to delete a topic I need only remove
>> it's
>> > data log directory from each broker, and the restart the brokers?  Is
>> it ok
>> > if it's a rolling restart?
>> >
>> > For some reason I thought I also had to do something on zookeeper
>> servers
>> > too, to remove the topics.  Is that not true?
>> >
>> > Jason
>> >
>> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <neha.narkhede@gmail.com
>> > >wrote:
>> >
>> > > >> The topics get automagically created if they don't exist.  Will
>> > > that still exist in 0.8?
>> > >
>> > > In 0.8, this is controlled by the auto.create.topics.enable config on
>> the
>> > > brokers. If this is set to true, topics will be created when a topic
>> > > metadata request is sent for a new topic. This feature is provided to
>> aid
>> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
>> tools
>> > > copy data for almost all topics from one cluster to the other, so auto
>> > > creating topics makes it much easier to migrate data between clusters.
>> > >
>> > > >> What's the procedure for deleting topics in kafka 0.7.2?
>> > >
>> > > There is no official way to delete a topic. You have to delete the
>> data
>> > > directory and bounce the brokers to take note of that.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > >
>> > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <jbr@squareup.com>
>> > wrote:
>> > >
>> > > > Thanks Neha,
>> > > >
>> > > > I added a comment to that ticket.
>> > > >
>> > > > Another question, I'm not sure if the behavior is different in 0.8
>> from
>> > > > 0.7.  But currently, I'm not predeclaring topics, before sending
>> > messages
>> > > > to topics.  The topics get automagically created if they don't
>> exist.
>> > >  Will
>> > > > that still exist in 0.8?  If so, it should also be automatic (using
>> a
>> > > > default expiration for all such topics), that they automagically go
>> > away
>> > > > after no longer being in use.
>> > > >
>> > > > What's the procedure for deleting topics in kafka 0.7.2?
>> > > >
>> > > > Jason
>> > > >
>> > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
>> neha.narkhede@gmail.com
>> > > > >wrote:
>> > > >
>> > > > > We plan to include support for deleting topics before the official
>> > > > release.
>> > > > > The JIRA tracking this feature is
>> > > > > https://issues.apache.org/jira/browse/KAFKA-330
>> > > > >
>> > > > > This is mainly an admin tool and is proposed to be run manually.
I
>> > > think
>> > > > > your suggestion is to set some sort of expiration time on topics
>> so
>> > > when
>> > > > > all data gets garbage collected and the expiration time lapses,
>> the
>> > > topic
>> > > > > is automatically deleted.
>> > > > >
>> > > > > For this expiration to work, we need some mechanism of knowing
the
>> > size
>> > > > of
>> > > > > the data logs for that topic on all brokers. And only when its
>> empty
>> > > for
>> > > > > all partitions of that topic, we delete it. This requires some
API
>> > that
>> > > > > will expose the data size of a topic on a broker, which we don't
>> have
>> > > > right
>> > > > > now. It might be worth thinking about it. Do you mind posting
your
>> > > > > suggestions for this tool on that JIRA ?
>> > > > >
>> > > > > Thanks,
>> > > > > Neha
>> > > > >
>> > > > >
>> > > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <jbr@squareup.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > I suspect this is not currently supported, but it seems
to be
>> for
>> > us
>> > > a
>> > > > > real
>> > > > > > use case.
>> > > > > >
>> > > > > > If we have a topic that is no longer receiving messages,
and all
>> > > > messages
>> > > > > > have been removed from the brokers, after the
>> log_retention_hours
>> > has
>> > > > > > expired, I'd love to have it then automatically remove the
topic
>> > from
>> > > > > > kafka.
>> > > > > >
>> > > > > > If subsequently a new message arrives for that topic, it
could
>> just
>> > > > > > recreate the topic as originally, and continue on.
>> > > > > >
>> > > > > > Is this easy to implement?
>> > > > > > Has it been requested?
>> > > > > > Does it make sense?
>> > > > > > Should I file a jira?
>> > > > > >
>> > > > > > Currently, we had a test client that inadvertently created
>> 1000's
>> > of
>> > > > > > topics.  The job of deleting all these appears to not be
a
>> > cumbersome
>> > > > > > manual process (are there any quick links to the recommended
way
>> > for
>> > > > > doing
>> > > > > > this?).
>> > > > > >
>> > > > > > I am using 0.7.2, but will eventually upgrade to 0.8, once
it's
>> > > > released
>> > > > > to
>> > > > > > beta....
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Jason
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message