kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1755) Improve error handling in log cleaner
Date Thu, 04 Dec 2014 23:24:12 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234780#comment-14234780

Guozhang Wang commented on KAFKA-1755:

Here are my two cents:

1. At the end of the day, Kafka will have two types of topics, one type only accepts keyed
messages and log compaction is used; the other one accepts any message and log cleaning is
used. Those two types of topics never exchange, i.e. once a topic is created with one of the
two types, it will never change its type until deletion.

2. Compressed message will be supported with log compaction, which will de-serialize the message
set and re-serialize.

3. With these two points in mind, I would suggest for now:
  a. Broker reject non-keyed messages for compacted topics.
  b. Broker reject compressed messages for compacted topics (this will be lifted after KAFKA-1374
is checked in).
  c. With this, it should never happen that compactor thread encountering a non-keyed / compressed
(this will be lifted after KAFKA-1374); if it happens, this would be a FATAL error and we
should throw an exception and halt the server. It indicates some operations are needed and
there are some code fixes before it can be restarted.

> Improve error handling in log cleaner
> -------------------------------------
>                 Key: KAFKA-1755
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1755
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Joel Koshy
>            Assignee: Joel Koshy
>              Labels: newbie++
>             Fix For: 0.8.3
> The log cleaner is a critical process when using compacted topics.
> However, if there is any error in any topic (notably if a key is missing) then the cleaner
exits and all other compacted topics will also be adversely affected - i.e., compaction stops
across the board.
> This can be improved by just aborting compaction for a topic on any error and keep the
thread from exiting.
> Another improvement would be to reject messages without keys that are sent to compacted
topics although this is not enough by itself.

This message was sent by Atlassian JIRA

View raw message