kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominic Evans (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-3978) Cannot truncate to a negative offset (-1) exception at broker startup
Date Fri, 21 Jul 2017 21:22:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096887#comment-16096887
] 

Dominic Evans edited comment on KAFKA-3978 at 7/21/17 9:21 PM:
---------------------------------------------------------------

[~ijuma] we actually hit this issue the other day on 0.10.2 and it seemed to have more serious
repercussions than initially suggested here.

Because the truncateTo exception causes kafka.server.ReplicaManager.makeFollowers to exit,
having removed any fetchers for the set of partitions passed in, we seem to end up with the
broker never following/replicating those partitions again unless the "bad" partition (seemingly
with an unknown checkpoint offset) is manually removed from ZK and the broker is restarted.


I know that KIP-101 reworked this area, so it may no longer be possible under 0.11 onwards.
I also wasn't entirely sure how we got into the state and was unable to reproduce it by manually
fiddling with replication-offset-checkpoint or otherwise.


was (Author: dnwe):
[~ijuma] we actually hit this issue the other day on 0.10.2 and it seemed to have more serious
repercussions than initially suggested here.

Because the truncateTo exception causes kafka.server.ReplicaManager.makeFollowers to exit,
having removed any fetchers for the set of partitions passed in, we seem to end up with the
broker never following/replicating those partitions again unless the "bad" partition (seemingly
with an unknown checkpoint offset) is manually removed from ZK and the broker is restarted.


> Cannot truncate to a negative offset (-1) exception at broker startup
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-3978
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3978
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.0.0
>         Environment: 3.13.0-87-generic 
>            Reporter: Juho Mäkinen
>            Priority: Critical
>              Labels: reliability, startup
>
> During broker startup sequence the broker server.log has this exception. Problem persists
after multiple restarts and also on another broker in the cluster.
> {code}
> INFO [Socket Server on Broker 1002], Started 1 acceptor threads (kafka.network.SocketServer)
> INFO [Socket Server on Broker 1002], Started 1 acceptor threads (kafka.network.SocketServer)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [ExpirationReaper-1002], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> INFO [GroupCoordinator 1002]: Starting up. (kafka.coordinator.GroupCoordinator)
> INFO [GroupCoordinator 1002]: Starting up. (kafka.coordinator.GroupCoordinator)
> INFO [GroupCoordinator 1002]: Startup complete. (kafka.coordinator.GroupCoordinator)
> INFO [GroupCoordinator 1002]: Startup complete. (kafka.coordinator.GroupCoordinator)
> INFO [Group Metadata Manager on Broker 1002]: Removed 0 expired offsets in 9 milliseconds.
(kafka.coordinator.GroupMetadataManager)
> INFO [Group Metadata Manager on Broker 1002]: Removed 0 expired offsets in 9 milliseconds.
(kafka.coordinator.GroupMetadataManager)
> INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
> INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
> INFO Creating /brokers/ids/1002 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
> INFO Creating /brokers/ids/1002 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
> INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
> INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
> INFO Registered broker 1002 at path /brokers/ids/1002 with addresses: PLAINTEXT ->
EndPoint(172.16.2.22,9092,PLAINTEXT) (kafka.utils.ZkUtils)
> INFO Registered broker 1002 at path /brokers/ids/1002 with addresses: PLAINTEXT ->
EndPoint(172.16.2.22,9092,PLAINTEXT) (kafka.utils.ZkUtils)
> INFO Kafka version : 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
> INFO Kafka commitId : b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
> INFO [Kafka Server 1002], started (kafka.server.KafkaServer)
> INFO [Kafka Server 1002], started (kafka.server.KafkaServer)
> Error when handling request {controller_id=1004,controller_epoch=1,partition_states=[..REALLY
LONG OUTPUT SNIPPED AWAY..], live_leaders=[{id=1004,host=172.16.6.187,port=9092},{id=1003,host=172.16.2.21,port=9092}]}
(kafka.server.KafkaApis)
> ERROR java.lang.IllegalArgumentException: Cannot truncate to a negative offset (-1).
>         at kafka.log.Log.truncateTo(Log.scala:731)
>         at kafka.log.LogManager$$anonfun$truncateTo$2.apply(LogManager.scala:288)
>         at kafka.log.LogManager$$anonfun$truncateTo$2.apply(LogManager.scala:280)
>         at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>         at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
>         at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
>         at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
>         at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
>         at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
>         at kafka.log.LogManager.truncateTo(LogManager.scala:280)
>         at kafka.server.ReplicaManager.makeFollowers(ReplicaManager.scala:802)
>         at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:648)
>         at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:144)
>         at kafka.server.KafkaApis.handle(KafkaApis.scala:80)
>         at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>         at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message