kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Frank Lyaruu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4311) Exception in NamedCache.flush - Key found in dirty key set, but entry is null
Date Wed, 19 Oct 2016 15:41:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589074#comment-15589074

Frank Lyaruu commented on KAFKA-4311:

I'm working on it, and frankly I'm at a loss.

I've made a repository here: https://github.com/flyaruu/kafka4311.git

I've isolated the code that does the joining. If I run this against my customer data (which
is person data I'm afraid I cannot share) I run into this bug pretty quickly, especially when
I increase the number of threads. If I set the CACHE_MAX_BYTES_BUFFERING_CONFIG to 0, I don't
see it.

I've tried to create a synthetic dataset (The code is in the GenerateTestData class), but
I've been unable to reproduce the bug with that dataset.

It seems to be data related, but at the same time the cache and threading settings seem to
have influence.

So either there is a bug that only manifests with very specific data, or there is a bug in
my code that somehow gets sidestepped when the cache is off.

Any ideas? Is there something obvious I'm doing wrong?

regards, Frank

> Exception in NamedCache.flush -  Key found in dirty key set, but entry is null 
> -------------------------------------------------------------------------------
>                 Key: KAFKA-4311
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4311
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions:
>            Reporter: Damian Guy
>            Assignee: Damian Guy
>             Fix For:
> Reported on the mailing list. Needs looking into how it could get in this state.
> [StreamThread-1] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [StreamThread-1] Failed to close state manager for StreamTask 0_0:
> org.apache.kafka.streams.errors.ProcessorStateException: task [0_0] Failed
> to close state store addr-organization
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:342)
> at
> org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(AbstractTask.java:121)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$2.apply(StreamThread.java:341)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.performOnAllTasks(StreamThread.java:322)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.closeAllStateManagers(StreamThread.java:338)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:299)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:262)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:245)
> Caused by: java.lang.IllegalStateException: Key found in dirty key set, but
> entry is null
> at
> org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:112)
> at
> org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:100)
> at
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:111)
> at
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.close(CachingKeyValueStore.java:117)
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:340)
> ... 7 more

This message was sent by Atlassian JIRA

View raw message