kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaikiran Pai <jai.forums2...@gmail.com>
Subject Re: Improve default Kafka logger settings to prevent extremely high disk space usage issue?
Date Thu, 15 Dec 2016 12:30:09 GMT


On Tuesday 13 December 2016 03:29 PM, Ismael Juma wrote:
> The log config settings for the controller and state change logger have
> been that way since they were introduced. They're generally useful when
> investigating issues with the controller. Looks like this is too noisy in
> some scenarios though. It may be worth filing a JIRA with specifics of your
> use case to see if something can be done to improve that.
For now, we decided to use a custom log4j config. I'll get back to this 
and check if there are specific log messages that cause this or if it's 
just regular logs at TRACE level which contribute to this, before filing 
a JIRA.

-Jaikiran



>
> Ismael
>
> On Tue, Dec 13, 2016 at 9:32 AM, Jaikiran Pai <jai.forums2013@gmail.com>
> wrote:
>
>> We happened to run into a disk space usage issue with Kafka 0.10.0.1 (the
>> version we are using) on one of our production setups this morning. Turns
>> out (log4j) logging from Kafka ended up using 81G and more of disk space.
>> Looking at the files, I see the controller.log itself is 30G and more (for
>> a single day). Looking at the default log4j.properties that's shipped in
>> Kafka, it uses the DailyRollingFileAppender which is one of the things that
>> contributes to this issue. I see that there's already a patch and JIRA to
>> fix this https://issues.apache.org/jira/browse/KAFKA-2394. It's been
>> marked for 0.11 because there wasn't a clear decision when to ship it.
>>
>> Given that we have been going through 0.10.x releases these days and the
>> 0.11 release looking a bit away, is there any chance, this specific JIRA
>> can make it to 0.10.x? I personally don't see any compatibility issues that
>> it will introduce when it comes to *functionality/features* of Kafka
>> itself, so I am not sure if it's that big a change to wait all the way till
>> 0.11. Furthermore, since the default shipped setting can cause issues like
>> the one I noted, I think it probably would be fine to include it in one of
>> the 0.10.x releases. Of course, we ourselves can setup the logging config
>> on our setup to use a size based rolling file config instead of the one
>> shipped by default, but if this is something that can make it to 0.10.x, we
>> would like to avoid doing this customization ourselves.
>>
>> That's one part of the issue. The other is, I see this in the default
>> shipped log4j.properties:
>>
>>
>> log4j.logger.kafka.controller=*TRACE,* controllerAppender
>> log4j.additivity.kafka.controller=false
>>
>> log4j.logger.state.change.logger=*TRACE*, stateChangeAppender
>> log4j.additivity.state.change.logger=false
>>
>>
>> Is it intentional to have this at TRACE level for the default shipped
>> config instead of having something like INFO or maybe DEBUG?
>>
>>
>> -Jaikiran
>>


Mime
View raw message