kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: auto-delete empty, old topics?
Date Thu, 14 Mar 2013 21:39:06 GMT
Jason,

Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
the Kafka log files from the server and bounced the broker. This should've
ideally deleted those topics from zookeeper, the consumer reads the same
zookeeper paths that the broker writes. Doing this should cause rebalancing
on the consumer and it should readjust its consumption to include the
remaining topics. But looks like that didn't happen somehow ?

I agree that log line is redundant, I think it is removed in 0.8

Thanks,
Neha


On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <jbr@squareup.com> wrote:

> Also,
>
> I see a bazillion consecutive log lines like this:
>
> 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> Clearing the current data chunk for this consumer iterator
>
> With the same content (not sure how useful that is!).
>
> Jason
>
> On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <jbr@squareup.com> wrote:
>
> > Hi Neha,
> >
> > So I did this, but I still see the full list of topics (most of which
> have
> > been deleted), in the consumer logs, e.g.:
> >
> > consumer.ZookeeperConsumerConnector -
> > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> > consume = List(<a bazillion topics, most of which have been deleted
> listed
> > here>)
> >
> > I select topics using the white list topic selector (using a wild-card
> > pattern, etc.).
> >
> > Thoughts?
> >
> > Does zookeeper keep separate info for the consumer, that needs to be
> > removed?
> >
> > Jason
> >
> > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> In Kafka 0.7.2, data log directory delete and then rolling restart will
> >> work to get rid of the topic completely from zookeeper as well as the
> >> broker.
> >> This is because the broker registers the topics and partitions from its
> >> data directory into zookeeper on startup.
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <jbr@squareup.com>
> wrote:
> >>
> >> > Thanks Neha,
> >> >
> >> > So are you saying that on 0.7.2, to delete a topic I need only remove
> >> it's
> >> > data log directory from each broker, and the restart the brokers?  Is
> >> it ok
> >> > if it's a rolling restart?
> >> >
> >> > For some reason I thought I also had to do something on zookeeper
> >> servers
> >> > too, to remove the topics.  Is that not true?
> >> >
> >> > Jason
> >> >
> >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> >> > >wrote:
> >> >
> >> > > >> The topics get automagically created if they don't exist.
 Will
> >> > > that still exist in 0.8?
> >> > >
> >> > > In 0.8, this is controlled by the auto.create.topics.enable config
> on
> >> the
> >> > > brokers. If this is set to true, topics will be created when a topic
> >> > > metadata request is sent for a new topic. This feature is provided
> to
> >> aid
> >> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> >> tools
> >> > > copy data for almost all topics from one cluster to the other, so
> auto
> >> > > creating topics makes it much easier to migrate data between
> clusters.
> >> > >
> >> > > >> What's the procedure for deleting topics in kafka 0.7.2?
> >> > >
> >> > > There is no official way to delete a topic. You have to delete the
> >> data
> >> > > directory and bounce the brokers to take note of that.
> >> > >
> >> > > Thanks,
> >> > > Neha
> >> > >
> >> > >
> >> > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <jbr@squareup.com>
> >> > wrote:
> >> > >
> >> > > > Thanks Neha,
> >> > > >
> >> > > > I added a comment to that ticket.
> >> > > >
> >> > > > Another question, I'm not sure if the behavior is different in
0.8
> >> from
> >> > > > 0.7.  But currently, I'm not predeclaring topics, before sending
> >> > messages
> >> > > > to topics.  The topics get automagically created if they don't
> >> exist.
> >> > >  Will
> >> > > > that still exist in 0.8?  If so, it should also be automatic
> (using
> >> a
> >> > > > default expiration for all such topics), that they automagically
> go
> >> > away
> >> > > > after no longer being in use.
> >> > > >
> >> > > > What's the procedure for deleting topics in kafka 0.7.2?
> >> > > >
> >> > > > Jason
> >> > > >
> >> > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
> >> neha.narkhede@gmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > We plan to include support for deleting topics before the
> official
> >> > > > release.
> >> > > > > The JIRA tracking this feature is
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-330
> >> > > > >
> >> > > > > This is mainly an admin tool and is proposed to be run
> manually. I
> >> > > think
> >> > > > > your suggestion is to set some sort of expiration time on
topics
> >> so
> >> > > when
> >> > > > > all data gets garbage collected and the expiration time
lapses,
> >> the
> >> > > topic
> >> > > > > is automatically deleted.
> >> > > > >
> >> > > > > For this expiration to work, we need some mechanism of knowing
> the
> >> > size
> >> > > > of
> >> > > > > the data logs for that topic on all brokers. And only when
its
> >> empty
> >> > > for
> >> > > > > all partitions of that topic, we delete it. This requires
some
> API
> >> > that
> >> > > > > will expose the data size of a topic on a broker, which
we don't
> >> have
> >> > > > right
> >> > > > > now. It might be worth thinking about it. Do you mind posting
> your
> >> > > > > suggestions for this tool on that JIRA ?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Neha
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <
> jbr@squareup.com
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > I suspect this is not currently supported, but it seems
to be
> >> for
> >> > us
> >> > > a
> >> > > > > real
> >> > > > > > use case.
> >> > > > > >
> >> > > > > > If we have a topic that is no longer receiving messages,
and
> all
> >> > > > messages
> >> > > > > > have been removed from the brokers, after the
> >> log_retention_hours
> >> > has
> >> > > > > > expired, I'd love to have it then automatically remove
the
> topic
> >> > from
> >> > > > > > kafka.
> >> > > > > >
> >> > > > > > If subsequently a new message arrives for that topic,
it could
> >> just
> >> > > > > > recreate the topic as originally, and continue on.
> >> > > > > >
> >> > > > > > Is this easy to implement?
> >> > > > > > Has it been requested?
> >> > > > > > Does it make sense?
> >> > > > > > Should I file a jira?
> >> > > > > >
> >> > > > > > Currently, we had a test client that inadvertently
created
> >> 1000's
> >> > of
> >> > > > > > topics.  The job of deleting all these appears to not
be a
> >> > cumbersome
> >> > > > > > manual process (are there any quick links to the recommended
> way
> >> > for
> >> > > > > doing
> >> > > > > > this?).
> >> > > > > >
> >> > > > > > I am using 0.7.2, but will eventually upgrade to 0.8,
once
> it's
> >> > > > released
> >> > > > > to
> >> > > > > > beta....
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Jason
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message