kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ismael Juma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3689) Exception when attempting to decrease connection count for address with no connections
Date Wed, 16 Aug 2017 10:42:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ismael Juma updated KAFKA-3689:
-------------------------------
    Fix Version/s:     (was: 0.11.0.1)
                   0.11.0.2

> Exception when attempting to decrease connection count for address with no connections
> --------------------------------------------------------------------------------------
>
>                 Key: KAFKA-3689
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3689
>             Project: Kafka
>          Issue Type: Bug
>          Components: network
>    Affects Versions: 0.9.0.1
>         Environment: ubuntu 14.04,
> java version "1.7.0_95"
> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.2)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
> 3 broker cluster (all 3 servers identical -  Intel Xeon E5-2670 @2.6GHz, 8cores, 16 threads
64 GB RAM & 1 TB Disk)
> Kafka Cluster is managed by 3 server ZK cluster (these servers are different from Kafka
broker servers). All 6 servers are connected via 10G switch. 
> Producers run from external servers.
>            Reporter: Buvaneswari Ramanan
>            Assignee: Ryan P
>             Fix For: 0.11.0.2
>
>         Attachments: kafka-3689-instrumentation.patch, KAFKA-3689.log.redacted
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> As per Ismael Juma's suggestion in email thread to users@kafka.apache.org with the same
subject, I am creating this bug report.
> The following error occurs in one of the brokers in our 3 broker cluster, which serves
about 8000 topics. These topics are single partitioned with a replication factor = 3. Each
topic gets data at a low rate  – 200 bytes per sec.  Leaders are balanced across the topics.
> Producers run from external servers (4 Ubuntu servers with same config as the brokers),
each producing to 2000 topics utilizing kafka-python library.
> This error message occurs repeatedly in one of the servers. Between the hours of 10:30am
and 1:30pm on 5/9/16, there were about 10 Million such occurrences. This was right after a
cluster restart.
> This is not the first time we got this error in this broker. In those instances, error
occurred hours / days after cluster restart.
> =====================================================
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count for address
with no connections, address: /X.Y.Z.144 (actual network address masked)
>         at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
>         at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
>         at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
>         at scala.collection.AbstractMap.getOrElse(Map.scala:59)
>         at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
>         at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
>         at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at kafka.network.Processor.run(SocketServer.scala:445)
>         at java.lang.Thread.run(Thread.java:745)
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count for address
with no connections, address: /X.Y.Z.144
>         at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
>         at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
>         at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
>         at scala.collection.AbstractMap.getOrElse(Map.scala:59)
>         at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
>         at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
>         at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at kafka.network.Processor.run(SocketServer.scala:445)
>         at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message