kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikumar Reddy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-1749) Brokers continually throw exceptions when there are hundreds of topic being fetched by mirrormaker
Date Fri, 07 Nov 2014 12:20:34 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Manikumar Reddy updated KAFKA-1749:
-----------------------------------
    Attachment: KAFKA-1749.patch

> Brokers continually throw exceptions when there are hundreds of topic being fetched by
mirrormaker
> --------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-1749
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1749
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.1.1, 0.8.2
>            Reporter: Min Zhou
>         Attachments: KAFKA-1749.patch
>
>
> Here is one piece of  millions of the exceptions. 
> {noformat}
> kafka.common.KafkaException: This operation cannot be completed on a complete request.
> at kafka.network.Transmission$class.expectIncomplete(Transmission.scala:34)
> at kafka.api.FetchResponseSend.expectIncomplete(FetchResponse.scala:191)
> at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:214)
> at kafka.network.Processor.write(SocketServer.scala:375)
> at kafka.network.Processor.run(SocketServer.scala:247)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> We use tools to hook function kafka.api.FetchResponseSend.writeTo, found 
>  fetchResponse.sizeInBytes was overflow.  Which means below code will get a result over
the limit of integer type
> {noformat}
> val sizeInBytes =
>     FetchResponse.headerSize +
>     dataGroupedByTopic.foldLeft(0) ((folded, curr) => {
>       val topicData = TopicData(curr._1, curr._2.map {
>         case (topicAndPartition, partitionData) => (topicAndPartition.partition, partitionData)
>       })
>       folded + topicData.sizeInBytes
>     })
> {noformat} 
>  If we just fetch few topics by mirrormaker, the brokers ran well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message