kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1499) Broker-side compression configuration
Date Wed, 01 Oct 2014 04:38:34 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154336#comment-14154336
] 

Jay Kreps commented on KAFKA-1499:
----------------------------------

Yeah I totally agree.

I agree that some heuristic that worked batch-by-batch might be okay, I hadn't thought of
that. Actually though I think the main motivation for this feature was to fix the compaction
issue, so if that is an okay fix just doing that would be an alternative.

I also agree that NoCompressionCodec should be the default and unless people know about the
change they will surely be confused by this switch. However I claim this is a temporary confusion
based on the fact that previously Kafka compression worked one way and now it will work a
new way. Plus they will in any case have this confusion if they turn on the feature. For any
new user the configuration docs will all be updated and in the process of learning how to
turn on compression they will learn how it works. I think we could help this with good release
notes (when doing an upgrade people always read that to ensure it is in-place compatible).

I guess in the end what I am arguing is that we should make a choice. Either a single compression
codec per topic is better and it should work that way or else having the producer specify
compression is better and it should work that way. Giving the user the choice seems nice but
it actually just adds complexity since now we will always have to document and explain both
and tell people about the configuration knob to choose and then advise them on how to best
make the choice (and then debug when they get lost in all this). If we think the right choice
is very situation specific (in situation x, chose broker.compression.enabled=true, in situation
y chose false) then okay maybe we need a config, but then let's figure out what the situations
you want one versus the other. If it isn't situation specific we should just choose one and
implement and document that.

> Broker-side compression configuration
> -------------------------------------
>
>                 Key: KAFKA-1499
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1499
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Joel Koshy
>            Assignee: Manikumar Reddy
>              Labels: newbie++
>             Fix For: 0.8.2
>
>         Attachments: KAFKA-1499.patch, KAFKA-1499.patch, KAFKA-1499_2014-08-15_14:20:27.patch,
KAFKA-1499_2014-08-21_21:44:27.patch, KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch,
KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch, KAFKA-1499_2014-09-25_11:05:57.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message