kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1590) Binarize trace level request logging along with debug level text logging
Date Wed, 01 Oct 2014 22:11:35 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155647#comment-14155647
] 

Guozhang Wang commented on KAFKA-1590:
--------------------------------------

[~jkreps] I agree that this would not be the default config. 

By default: the request log would be written in text.

When this knob turns on: the request log would be written only as summary in text (i.e. today's
DEBUG level logging), and the detailed log will be recorded as kafka-request-details.gzip/[any
other compression types?].

I personally do not know what the % savings would be with the proposed binary / compression
format compared the the text format. Would be great if we can get some numbers with a preliminary
patch.

[~Abhishek Sharma] do you think it is possible that we write a simple cross-platform tool
to parse the request-details binary files?

> Binarize trace level request logging along with debug level text logging
> ------------------------------------------------------------------------
>
>                 Key: KAFKA-1590
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1590
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Guozhang Wang
>            Assignee: Abhishek Sharma
>              Labels: newbie
>             Fix For: 0.9.0
>
>         Attachments: Jira-1590-Log4j Binary Appender.patch
>
>
> With trace level logging, the request handling logs can grow very fast depending on the
client behavior (e.g. consumer with 0 maxWait and hence keep sending fetch requests). Previously
we have changed it to debug level which only provides a summary of the requests, omitting
request details. However this does not work perfectly since summaries are not sufficient for
trouble-shooting, and turning on trace level upon issues will be too late.
> The proposed solution here, is to default to debug level logging with trace level logging
printed as binary format at the same time. The generated binary files can then be further
compressed / rolled out. When needed, we will then decompress / parse the trace logs into
texts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message