kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanna Gautam (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-5628) Kafka Startup fails on corrupted index files
Date Sun, 23 Jul 2017 16:16:02 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanna Gautam updated KAFKA-5628:
-----------------------------------
    Description: 
One of our kafka brokers shut down after a load test and while there are some corrupted index
files , the broker is failing to start with a unsafe memory access error


{code:java}
[2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare to
shutdown (kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled
Java code
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53)
        at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854)
        at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827)
        at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136)
        at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149)
        at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225)
        at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.log.LogSegment.recover(LogSegment.scala:224)
        at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
        at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.Log.loadSegments(Log.scala:188)
        at kafka.log.Log.<init>(Log.scala:116)
        at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}

This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because these
topics are actively in use and the other empty indices are recovered fine..

Kafka on the machine had died because the disk was full. 
It seems to have resolved after the disk issue. Should kafka just check disk at startup and
refuse to continue starting up? 

  was:
One of our kafka brokers shut down after a load test and while there are some corrupted index
files , the broker is failing to start with a unsafe memory access error


{code:java}
[2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare to
shutdown (kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled
Java code
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53)
        at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854)
        at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827)
        at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136)
        at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149)
        at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225)
        at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.log.LogSegment.recover(LogSegment.scala:224)
        at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
        at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.Log.loadSegments(Log.scala:188)
        at kafka.log.Log.<init>(Log.scala:116)
        at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}

This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because these
topics are actively in use and the other empty indices are recovered fine..

It seems the machine had died because the disk was full. 
It seems to have resolved after the disk issue. Should kafka just check disk at startup and
refuse to continue starting up? 


> Kafka Startup fails on corrupted index files
> --------------------------------------------
>
>                 Key: KAFKA-5628
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5628
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.2.0
>         Environment: Ubuntu 14.04, Java 8(1.8.0_65)
>            Reporter: Prasanna Gautam
>            Assignee: Jun Rao
>            Priority: Minor
>
> One of our kafka brokers shut down after a load test and while there are some corrupted
index files , the broker is failing to start with a unsafe memory access error
> {code:java}
> [2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare
to shutdown (kafka.server.KafkaServerStartable)
> java.lang.InternalError: a fault occurred in a recent unsafe memory access operation
in compiled Java code
>         at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53)
>         at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854)
>         at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827)
>         at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136)
>         at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149)
>         at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225)
>         at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:893)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
>         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at kafka.log.LogSegment.recover(LogSegment.scala:224)
>         at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
>         at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
>         at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>         at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>         at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>         at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
>         at kafka.log.Log.loadSegments(Log.scala:188)
>         at kafka.log.Log.<init>(Log.scala:116)
>         at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
>         at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because
these topics are actively in use and the other empty indices are recovered fine..
> Kafka on the machine had died because the disk was full. 
> It seems to have resolved after the disk issue. Should kafka just check disk at startup
and refuse to continue starting up? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message