kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ismael Juma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3917) Some __consumer_offsets replicas grow way too big
Date Mon, 30 Oct 2017 12:10:02 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ismael Juma updated KAFKA-3917:
-------------------------------
    Affects Version/s:     (was: 0.8.2.2)
                       0.10.2.0

> Some __consumer_offsets replicas grow way too big
> -------------------------------------------------
>
>                 Key: KAFKA-3917
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3917
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.10.2.0
>         Environment: Runs with Docker 1.10.1 in a container on 
> Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64
>            Reporter: Maxim Vladimirskiy
>              Labels: reliability
>
> We noticed that some replicas of partitions of the __consumer_offsets topic grow way
too big. Looking inside respective folders it became apparent that old segments had not been
cleaned up. Please see below example of disk usage data for both affected and not affected
partitions:
> Not affected partitions:
>     Partition: 0	Leader: 2	Replicas: 2,3,4	Isr: 2,4,3
>         2: 49M
>         3: 49M
>         4: 49M
> Affected partitions:
>     Partition: 10	Leader: 2	Replicas: 2,0,1	Isr: 1,2,0
>         0: 86M
>         1: 22G <<< too big!
>         2: 86M
>     Partition: 38	Leader: 0	Replicas: 0,4,1	Isr: 1,0,4
>         0: 43M
>         1: 26G <<<  too big!
>         4: 26G <<<  too big!
> As you can see sometimes only one replica is affected, sometimes both replicas are affected.
> When I try to restart a broker that has affected replicas it fails to start with an exception
that looks like this:
> [2016-06-28 23:15:20,441] ERROR There was an error in one of the threads during logs
loading: java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index
file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero size but the
last offset is -676703869 and the base offset is 0 (kafka.log.LogManager)
> [2016-06-28 23:15:20,442] FATAL [Kafka Server 1], Fatal error during KafkaServer startup.
Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file
(/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero size but the last
offset is -676703869 and the base offset is 0
>         at scala.Predef$.require(Predef.scala:233)
>         at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
>         at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
>         at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at kafka.log.Log.loadSegments(Log.scala:183)
>         at kafka.log.Log.<init>(Log.scala:67)
>         at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
>         at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> After the content of the affected partition is deleted broker starts successfully. 
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message