kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rico (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-6059) Kafka cant delete old log files on windows
Date Fri, 13 Oct 2017 01:20:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

rico updated KAFKA-6059:
------------------------
    Description: 
I had a trouble for delete old log files.
Now , I had disk space usage issues.

I found exception from kafka's log and content as below:
kafka.common.KafkaStorageException: Failed to change the log file suffix from  to .deleted
for log segment 0
	at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:340)
	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:344)
	at kafka.log.Log.asyncDeleteSegment(Log.scala:981)
	at kafka.log.Log.deleteSegment(Log.scala:971)
	at kafka.log.Log.$anonfun$deleteOldSegments$1(Log.scala:673)
	at kafka.log.Log.$anonfun$deleteOldSegments$1$adapted(Log.scala:673)
	at scala.collection.immutable.List.foreach(List.scala:378)
	at kafka.log.Log.deleteOldSegments(Log.scala:673)
	at kafka.log.Log.deleteRetenionMsBreachedSegments(Log.scala:703)
	at kafka.log.Log.deleteOldSegments(Log.scala:697)
	at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:474)
	at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:472)
	at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
	at scala.collection.Iterator.foreach(Iterator.scala:929)
	at scala.collection.Iterator.foreach$(Iterator.scala:929)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
	at scala.collection.IterableLike.foreach(IterableLike.scala:71)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
	at kafka.log.LogManager.cleanupLogs(LogManager.scala:472)
	at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:200)
	at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
	at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log
-> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot
access the file because it is being used by another process.

	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:711)
	at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:210)
	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:342)
	... 29 more
	Suppressed: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log
-> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot
access the file because it is being used by another process.

		at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
		at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
		at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
		at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
		at java.nio.file.Files.move(Files.java:1395)
		at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:708)
		... 31 more

does anyone can fix it?

  was:
I had a trouble for delete old log files.
Now , I have disk space usage issues.

I found exception from kafka's log and content as below:
kafka.common.KafkaStorageException: Failed to change the log file suffix from  to .deleted
for log segment 0
	at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:340)
	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:344)
	at kafka.log.Log.asyncDeleteSegment(Log.scala:981)
	at kafka.log.Log.deleteSegment(Log.scala:971)
	at kafka.log.Log.$anonfun$deleteOldSegments$1(Log.scala:673)
	at kafka.log.Log.$anonfun$deleteOldSegments$1$adapted(Log.scala:673)
	at scala.collection.immutable.List.foreach(List.scala:378)
	at kafka.log.Log.deleteOldSegments(Log.scala:673)
	at kafka.log.Log.deleteRetenionMsBreachedSegments(Log.scala:703)
	at kafka.log.Log.deleteOldSegments(Log.scala:697)
	at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:474)
	at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:472)
	at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
	at scala.collection.Iterator.foreach(Iterator.scala:929)
	at scala.collection.Iterator.foreach$(Iterator.scala:929)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
	at scala.collection.IterableLike.foreach(IterableLike.scala:71)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
	at kafka.log.LogManager.cleanupLogs(LogManager.scala:472)
	at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:200)
	at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
	at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log
-> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot
access the file because it is being used by another process.

	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:711)
	at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:210)
	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:342)
	... 29 more
	Suppressed: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log
-> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot
access the file because it is being used by another process.

		at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
		at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
		at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
		at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
		at java.nio.file.Files.move(Files.java:1395)
		at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:708)
		... 31 more

does anyone can fix it?


> Kafka cant delete old log files on windows
> ------------------------------------------
>
>                 Key: KAFKA-6059
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6059
>             Project: Kafka
>          Issue Type: Bug
>          Components: log
>    Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.0
>         Environment: OS:windows 2016
> kafka:10.0.2.1
> zookeeper:3.5.2
>            Reporter: rico
>            Priority: Critical
>              Labels: windows
>
> I had a trouble for delete old log files.
> Now , I had disk space usage issues.
> I found exception from kafka's log and content as below:
> kafka.common.KafkaStorageException: Failed to change the log file suffix from  to .deleted
for log segment 0
> 	at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:340)
> 	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:344)
> 	at kafka.log.Log.asyncDeleteSegment(Log.scala:981)
> 	at kafka.log.Log.deleteSegment(Log.scala:971)
> 	at kafka.log.Log.$anonfun$deleteOldSegments$1(Log.scala:673)
> 	at kafka.log.Log.$anonfun$deleteOldSegments$1$adapted(Log.scala:673)
> 	at scala.collection.immutable.List.foreach(List.scala:378)
> 	at kafka.log.Log.deleteOldSegments(Log.scala:673)
> 	at kafka.log.Log.deleteRetenionMsBreachedSegments(Log.scala:703)
> 	at kafka.log.Log.deleteOldSegments(Log.scala:697)
> 	at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:474)
> 	at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:472)
> 	at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
> 	at scala.collection.Iterator.foreach(Iterator.scala:929)
> 	at scala.collection.Iterator.foreach$(Iterator.scala:929)
> 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
> 	at scala.collection.IterableLike.foreach(IterableLike.scala:71)
> 	at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
> 	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> 	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
> 	at kafka.log.LogManager.cleanupLogs(LogManager.scala:472)
> 	at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:200)
> 	at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 	at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log
-> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot
access the file because it is being used by another process.
> 	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
> 	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> 	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
> 	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> 	at java.nio.file.Files.move(Files.java:1395)
> 	at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:711)
> 	at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:210)
> 	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:342)
> 	... 29 more
> 	Suppressed: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log
-> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot
access the file because it is being used by another process.
> 		at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
> 		at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> 		at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> 		at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> 		at java.nio.file.Files.move(Files.java:1395)
> 		at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:708)
> 		... 31 more
> does anyone can fix it?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message