kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seweryn Habdank-Wojewodzki (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
Date Mon, 11 Sep 2017 15:02:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161405#comment-16161405
] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-4392 at 9/11/17 3:01 PM:
----------------------------------------------------------------------------

I have the same problem. num.stream.threads = 4, number partitions = 10, replication factor
= 2.
At some moment KStream based app ends with:

ERROR StateDirectory:227 - stream-thread [...] Failed to lock the state directory due to an
unexpected exception

java.nio.file.DirectoryNotEmptyException: /data/myapp/tmp/kafka-state/streamer/18_4
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) ~[?:1.8.0_102]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[?:1.8.0_102]
        at java.nio.file.Files.delete(Files.java:1126) ~[?:1.8.0_102]
        at org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:597) ~[myapp-streamer.jar:?]
        at org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:580) ~[myapp-streamer.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2688) ~[?:1.8.0_102]
        at java.nio.file.Files.walkFileTree(Files.java:2742) ~[?:1.8.0_102]
        at org.apache.kafka.common.utils.Utils.delete(Utils.java:580) ~[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:221)
[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:878)
[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:569)
[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
[myapp-streamer.jar:?]



was (Author: habdank):
I have the same problem. num.stream.threads = 4 in number partitions = 10.
At some moment KStream based app ends with:

ERROR StateDirectory:227 - stream-thread [...] Failed to lock the state directory due to an
unexpected exception

java.nio.file.DirectoryNotEmptyException: /data/myapp/tmp/kafka-state/streamer/18_4
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) ~[?:1.8.0_102]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[?:1.8.0_102]
        at java.nio.file.Files.delete(Files.java:1126) ~[?:1.8.0_102]
        at org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:597) ~[myapp-streamer.jar:?]
        at org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:580) ~[myapp-streamer.jar:?]
        at java.nio.file.Files.walkFileTree(Files.java:2688) ~[?:1.8.0_102]
        at java.nio.file.Files.walkFileTree(Files.java:2742) ~[?:1.8.0_102]
        at org.apache.kafka.common.utils.Utils.delete(Utils.java:580) ~[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:221)
[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:878)
[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:569)
[myapp-streamer.jar:?]
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
[myapp-streamer.jar:?]


> Failed to lock the state directory due to an unexpected exception
> -----------------------------------------------------------------
>
>                 Key: KAFKA-4392
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4392
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.10.1.0
>            Reporter: Ara Ebrahimi
>            Assignee: Guozhang Wang
>             Fix For: 0.10.2.0
>
>
> This happened on streaming startup, on a clean installation, no existing folder. Here
I was starting 4 instances of our streaming app on 4 machines and one threw this exception.
Seems to me there’s a race condition somewhere when instances discover others, or something
like that.
> 2016-11-02 15:43:47 INFO  StreamRunner:59 - Started http server successfully.
> 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state directory due
to an unexpected exception
> java.nio.file.NoSuchFileException: /data/1/kafka-streams/myapp-streams/7_21/.lock
> 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
> 	at java.nio.channels.FileChannel.open(FileChannel.java:287)
> 	at java.nio.channels.FileChannel.open(FileChannel.java:335)
> 	at org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176)
> 	at org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90)
> 	at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140)
> 	at org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552)
> 	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459)
> 	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
> ^C
> [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/
> ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or directory
> [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/
> total 4
> drwxr-xr-x 74 root root 4096 Nov  2 15:44 .
> drwxr-xr-x  3 root root   27 Nov  2 15:43 ..
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_1
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_13
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_14
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_16
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_2
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_22
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_28
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_3
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_31
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_5
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_7
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_8
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_9
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_1
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_10
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_14
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_15
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_16
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_17
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_18
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_3
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_5
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_1
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_10
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_12
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_20
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_24
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_10
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_11
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_19
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_20
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_25
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_26
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_3
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_11
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_12
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_18
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_19
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_24
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_25
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_26
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_4
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_9
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_1
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_10
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_11
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_13
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_15
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_17
> drwxr-xr-x  3 root root   58 Nov  2 15:43 5_18
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_13
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_15
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_18
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_19
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_20
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_21
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_24
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_4
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_7
> drwxr-xr-x  2 root root   18 Nov  2 15:43 6_8
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_16
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_19
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_20
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_25
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_28
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_31
> drwxr-xr-x  2 root root   18 Nov  2 15:43 7_7
> drwxr-xr-x  2 root root   18 Nov  2 15:43 8_13
> drwxr-xr-x  2 root root   18 Nov  2 15:43 8_21
> drwxr-xr-x  2 root root   18 Nov  2 15:43 8_23
> drwxr-xr-x  2 root root   18 Nov  2 15:43 8_26
> drwxr-xr-x  2 root root   18 Nov  2 15:43 8_31



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message