kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mickael Maison (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3239) Timing issue in controller metrics on topic delete
Date Thu, 13 Jul 2017 10:34:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085523#comment-16085523
] 

Mickael Maison commented on KAFKA-3239:
---------------------------------------

We are still seeing this with 0.10.2.1:
{quote}
[2017-04-19 14:56:26,251] ERROR Error printing regular metrics: (com.airbnb.metrics.StatsDReporter)
java.util.NoSuchElementException: key not found: [TOPIC_NAME_REDACTED,0]
    at scala.collection.MapLike$class.default(MapLike.scala:228)
    at scala.collection.AbstractMap.default(Map.scala:59)
    at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
    at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:210)
    at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:208)
    at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:114)
    at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:113)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
    at scala.collection.TraversableOnce$class.count(TraversableOnce.scala:113)
    at scala.collection.AbstractTraversable.count(Traversable.scala:104)
    at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply$mcI$sp(KafkaController.scala:208)
    at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
    at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
    at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:204)
    at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:202)
    at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:163)
    at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:37)
    at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28)
    at com.airbnb.metrics.StatsDReporter.sendAMetric(StatsDReporter.java:131)
    at com.airbnb.metrics.StatsDReporter.sendAllKafkaMetrics(StatsDReporter.java:119)
    at com.airbnb.metrics.StatsDReporter.run(StatsDReporter.java:85)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:483)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:316)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:190)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
    at java.lang.Thread.run(Thread.java:809)
{quote}

The topic was stuck in "Marked for deletion" state. Such topics don't have leaders, so it's
probably the issue is probably with:
{quote}
controllerContext.partitionLeadershipInfo(topicPartition)
{quote} 

We're trying out a fix at the moment, if that works I'll send a PR.

> Timing issue in controller metrics on topic delete
> --------------------------------------------------
>
>                 Key: KAFKA-3239
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3239
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller
>    Affects Versions: 0.9.0.0
>            Reporter: Rajini Sivaram
>            Assignee: Rajini Sivaram
>
> Noticed this exception in our logs:
> {quote}
> java.util.NoSuchElementException: key not found: [sometopic,0]
>         at scala.collection.MapLike$class.default(MapLike.scala:228)
>         at scala.collection.AbstractMap.default(Map.scala:59)
>         at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
>         at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:209)
>         at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:208)
>         at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:114)
>         at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:113)
>         at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>         at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>         at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
>         at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
>         at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
>         at scala.collection.TraversableOnce$class.count(TraversableOnce.scala:113)
>         at scala.collection.AbstractTraversable.count(Traversable.scala:104)
>         at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply$mcI$sp(KafkaController.scala:208)
>         at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
>         at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
>         at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>         at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:204)
>         at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:202)
>         at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:163)
>         at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:37)
>         at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28)
>         at com.airbnb.metrics.StatsDReporter.sendAMetric(StatsDReporter.java:131)
>         at com.airbnb.metrics.StatsDReporter.sendAllKafkaMetrics(StatsDReporter.java:119)
>         at com.airbnb.metrics.StatsDReporter.run(StatsDReporter.java:85)
> {quote}
> The exception indicates that the topic was in {{controllerContext.partitionReplicaAssignment}}
but not in {{controllerContext.partitionLeadershipInfo}}. This can occur during {{KafkaController.removeTopic()}}
since it is not synchronized.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message