kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: auto-delete empty, old topics?
Date Thu, 14 Mar 2013 22:20:08 GMT
Yes, you're description matches what I did.  And the brokers have been
bounced many times since then (they are auto-deployed many times a day,
etc.).  And the consumers also have been restarted many times since then.

Could it be related to using the white-list topic selector, etc.?

Let me know if you need me to look a the state of my zookeeper data....

Jason

On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Jason,
>
> Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
> the Kafka log files from the server and bounced the broker. This should've
> ideally deleted those topics from zookeeper, the consumer reads the same
> zookeeper paths that the broker writes. Doing this should cause rebalancing
> on the consumer and it should readjust its consumption to include the
> remaining topics. But looks like that didn't happen somehow ?
>
> I agree that log line is redundant, I think it is removed in 0.8
>
> Thanks,
> Neha
>
>
> On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <jbr@squareup.com> wrote:
>
> > Also,
> >
> > I see a bazillion consecutive log lines like this:
> >
> > 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> > Clearing the current data chunk for this consumer iterator
> >
> > With the same content (not sure how useful that is!).
> >
> > Jason
> >
> > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <jbr@squareup.com>
> wrote:
> >
> > > Hi Neha,
> > >
> > > So I did this, but I still see the full list of topics (most of which
> > have
> > > been deleted), in the consumer logs, e.g.:
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > I select topics using the white list topic selector (using a wild-card
> > > pattern, etc.).
> > >
> > > Thoughts?
> > >
> > > Does zookeeper keep separate info for the consumer, that needs to be
> > > removed?
> > >
> > > Jason
> > >
> > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > >wrote:
> > >
> > >> In Kafka 0.7.2, data log directory delete and then rolling restart
> will
> > >> work to get rid of the topic completely from zookeeper as well as the
> > >> broker.
> > >> This is because the broker registers the topics and partitions from
> its
> > >> data directory into zookeeper on startup.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >>
> > >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <jbr@squareup.com>
> > wrote:
> > >>
> > >> > Thanks Neha,
> > >> >
> > >> > So are you saying that on 0.7.2, to delete a topic I need only
> remove
> > >> it's
> > >> > data log directory from each broker, and the restart the brokers?
>  Is
> > >> it ok
> > >> > if it's a rolling restart?
> > >> >
> > >> > For some reason I thought I also had to do something on zookeeper
> > >> servers
> > >> > too, to remove the topics.  Is that not true?
> > >> >
> > >> > Jason
> > >> >
> > >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > >> The topics get automagically created if they don't exist.
 Will
> > >> > > that still exist in 0.8?
> > >> > >
> > >> > > In 0.8, this is controlled by the auto.create.topics.enable config
> > on
> > >> the
> > >> > > brokers. If this is set to true, topics will be created when
a
> topic
> > >> > > metadata request is sent for a new topic. This feature is provided
> > to
> > >> aid
> > >> > > with the 0.7->0.8 migration tool and the mirror maker tools.
These
> > >> tools
> > >> > > copy data for almost all topics from one cluster to the other,
so
> > auto
> > >> > > creating topics makes it much easier to migrate data between
> > clusters.
> > >> > >
> > >> > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > >> > >
> > >> > > There is no official way to delete a topic. You have to delete
the
> > >> data
> > >> > > directory and bounce the brokers to take note of that.
> > >> > >
> > >> > > Thanks,
> > >> > > Neha
> > >> > >
> > >> > >
> > >> > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <jbr@squareup.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > Thanks Neha,
> > >> > > >
> > >> > > > I added a comment to that ticket.
> > >> > > >
> > >> > > > Another question, I'm not sure if the behavior is different
in
> 0.8
> > >> from
> > >> > > > 0.7.  But currently, I'm not predeclaring topics, before
sending
> > >> > messages
> > >> > > > to topics.  The topics get automagically created if they
don't
> > >> exist.
> > >> > >  Will
> > >> > > > that still exist in 0.8?  If so, it should also be automatic
> > (using
> > >> a
> > >> > > > default expiration for all such topics), that they automagically
> > go
> > >> > away
> > >> > > > after no longer being in use.
> > >> > > >
> > >> > > > What's the procedure for deleting topics in kafka 0.7.2?
> > >> > > >
> > >> > > > Jason
> > >> > > >
> > >> > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
> > >> neha.narkhede@gmail.com
> > >> > > > >wrote:
> > >> > > >
> > >> > > > > We plan to include support for deleting topics before
the
> > official
> > >> > > > release.
> > >> > > > > The JIRA tracking this feature is
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-330
> > >> > > > >
> > >> > > > > This is mainly an admin tool and is proposed to be
run
> > manually. I
> > >> > > think
> > >> > > > > your suggestion is to set some sort of expiration time
on
> topics
> > >> so
> > >> > > when
> > >> > > > > all data gets garbage collected and the expiration
time
> lapses,
> > >> the
> > >> > > topic
> > >> > > > > is automatically deleted.
> > >> > > > >
> > >> > > > > For this expiration to work, we need some mechanism
of knowing
> > the
> > >> > size
> > >> > > > of
> > >> > > > > the data logs for that topic on all brokers. And only
when its
> > >> empty
> > >> > > for
> > >> > > > > all partitions of that topic, we delete it. This requires
some
> > API
> > >> > that
> > >> > > > > will expose the data size of a topic on a broker, which
we
> don't
> > >> have
> > >> > > > right
> > >> > > > > now. It might be worth thinking about it. Do you mind
posting
> > your
> > >> > > > > suggestions for this tool on that JIRA ?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Neha
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <
> > jbr@squareup.com
> > >> >
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > I suspect this is not currently supported, but
it seems to
> be
> > >> for
> > >> > us
> > >> > > a
> > >> > > > > real
> > >> > > > > > use case.
> > >> > > > > >
> > >> > > > > > If we have a topic that is no longer receiving
messages, and
> > all
> > >> > > > messages
> > >> > > > > > have been removed from the brokers, after the
> > >> log_retention_hours
> > >> > has
> > >> > > > > > expired, I'd love to have it then automatically
remove the
> > topic
> > >> > from
> > >> > > > > > kafka.
> > >> > > > > >
> > >> > > > > > If subsequently a new message arrives for that
topic, it
> could
> > >> just
> > >> > > > > > recreate the topic as originally, and continue
on.
> > >> > > > > >
> > >> > > > > > Is this easy to implement?
> > >> > > > > > Has it been requested?
> > >> > > > > > Does it make sense?
> > >> > > > > > Should I file a jira?
> > >> > > > > >
> > >> > > > > > Currently, we had a test client that inadvertently
created
> > >> 1000's
> > >> > of
> > >> > > > > > topics.  The job of deleting all these appears
to not be a
> > >> > cumbersome
> > >> > > > > > manual process (are there any quick links to the
recommended
> > way
> > >> > for
> > >> > > > > doing
> > >> > > > > > this?).
> > >> > > > > >
> > >> > > > > > I am using 0.7.2, but will eventually upgrade
to 0.8, once
> > it's
> > >> > > > released
> > >> > > > > to
> > >> > > > > > beta....
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Jason
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message