kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1755) Improve error handling in log cleaner
Date Thu, 04 Dec 2014 23:24:12 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234780#comment-14234780
] 

Guozhang Wang commented on KAFKA-1755:
--------------------------------------

Here are my two cents:

1. At the end of the day, Kafka will have two types of topics, one type only accepts keyed
messages and log compaction is used; the other one accepts any message and log cleaning is
used. Those two types of topics never exchange, i.e. once a topic is created with one of the
two types, it will never change its type until deletion.

2. Compressed message will be supported with log compaction, which will de-serialize the message
set and re-serialize.

3. With these two points in mind, I would suggest for now:
  a. Broker reject non-keyed messages for compacted topics.
  b. Broker reject compressed messages for compacted topics (this will be lifted after KAFKA-1374
is checked in).
  c. With this, it should never happen that compactor thread encountering a non-keyed / compressed
(this will be lifted after KAFKA-1374); if it happens, this would be a FATAL error and we
should throw an exception and halt the server. It indicates some operations are needed and
there are some code fixes before it can be restarted.

> Improve error handling in log cleaner
> -------------------------------------
>
>                 Key: KAFKA-1755
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1755
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Joel Koshy
>            Assignee: Joel Koshy
>              Labels: newbie++
>             Fix For: 0.8.3
>
>
> The log cleaner is a critical process when using compacted topics.
> However, if there is any error in any topic (notably if a key is missing) then the cleaner
exits and all other compacted topics will also be adversely affected - i.e., compaction stops
across the board.
> This can be improved by just aborting compaction for a topic on any error and keep the
thread from exiting.
> Another improvement would be to reject messages without keys that are sent to compacted
topics although this is not enough by itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message