kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6415) KafkaLog4jAppender deadlocks when logging from producer network thread
Date Tue, 02 Jan 2018 15:16:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308193#comment-16308193
] 

ASF GitHub Bot commented on KAFKA-6415:
---------------------------------------

rajinisivaram opened a new pull request #4375: KAFKA-6415: Avoid info log during metadata
update in network thread
URL: https://github.com/apache/kafka/pull/4375
 
 
   When a log entry is appended to a Kafka topic using `KafkaLog4jAppender`, the producer.send
operation may block waiting for metadata. This can result in deadlocks in a couple of scenarios
if a log entry from the producer network thread is also at a log level that results in the
entry being appended to a Kafka topic.
   1. Producer's network thread will attempt to send data to a Kafka topic and this is unsafe
since producer.send may block waiting for metadata, causing a deadlock since the thread will
not process the metadata request/response.
   2. `KafkaLog4jAppender#append` is invoked while holding the lock of the logger. So the
thread waiting for metadata in the initial send will be holding the logger lock. If the producer
network thread has.a log entry that needs to be appended, it will attempt to acquire the logger
lock and deadlock.
   
   This is a temporary workaround to avoid deadlocks in system tests by avoiding logs at INFO
level in the producer network thread during metadata update. The fix has been verified using
the system tests log4j_appender_test.py which started failing when the info-level log entry
was introduced.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> KafkaLog4jAppender deadlocks when logging from producer network thread
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-6415
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6415
>             Project: Kafka
>          Issue Type: Bug
>          Components: log
>            Reporter: Rajini Sivaram
>            Assignee: Rajini Sivaram
>             Fix For: 1.1.0
>
>
> When a log entry is appended to a Kafka topic using KafkaLog4jAppender, the producer.send
operation may block waiting for metadata. This can result in deadlocks in a couple of scenarios
if a log entry from the producer network thread is also at a log level that results in the
entry being appended to a Kafka topic.
> 1. Producer's network thread will attempt to send data to a Kafka topic and this is unsafe
since producer.send may block waiting for metadata, causing a deadlock since the thread will
not process the metadata request/response.
> 2. KafkaLog4jAppender#append is invoked while holding the lock of the logger. So the
thread waiting for metadata in the initial send will be holding the logger lock. If the producer
network thread has.a log entry that needs to be appended, it will attempt to acquire the logger
lock and deadlock.
> This was probably the case right from the beginning when KafkaLog4jAppender was introduced,
but did not cause any issues so far since there were only debug log entries in that path which
were not logged to a Kafka topic by any of the tests. A recent info level log entry introduced
by the commit https://github.com/apache/kafka/commit/a3aea3cf4dbedb293f2d7859e0298bebc8e2185f
is causing system test failures in log4j_appender_test.py due to the deadlock.
> The asynchronous append case can be fixed by moving all send operations to a separate
thread. But KafkaLog4jAppender also has a syncSend option which blocks append while holding
the logger lock until the send completes. Not sure how this can be fixed if we want to support
log appends from the producer network thread.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message