kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seweryn Habdank-Wojewodzki (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-5861) KStream close( withTimeout ) - does not work under load conditions in the multi-threaded KStream application
Date Tue, 12 Sep 2017 08:25:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162654#comment-16162654
] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5861 at 9/12/17 8:24 AM:
----------------------------------------------------------------------------

Property: num.stream.threads=4; 
number of partitions in the topic = 10; 
replication factor = 2

I see the following, when I do not send message at all, I can stop the app by using kill PID.
When I am sending messages, kill PID does not work at all. I need to use kill -9 PID.

To be precise, I had made shutdown hook and then I see log "Got shutdown request ..." and
nothing more.
If I will send single message and then wait quite a long time (more than 30 or 60 sec) then
it is probable kill PID will work.
When I am sending steady couple like 2-5 messages per second mostly (>90%) kill PID does
not work - I need to use kill -9 PID.
When I am sending like 1000 or more messages per second, then kill PID does not work at all
- again kill -9 PID was doing the job.
But I suppose kill -9 PID is not the way to go, or?

Core of my shutdown hook is below. The small comment - it is not important if I have System.exit(0)
or not, as the procedure never comes to this point :-).

        Runtime.getRuntime().addShutdownHook( new Thread( () -> {
            LOG.info( "Got shutdown requests ..." );
            synchronized ( this ) {
                streams.close( 5L, TimeUnit.SECONDS );
            }

            LOG.info( "Streamer is stopped." );
            System.exit( 0 );
        }
        ) );



was (Author: habdank):
Property: num.stream.threads=4; 
number of partitions in the topic = 10; 
replication factor = 2

I see the following, when I do not send message at all, I can stop the app by using kill PID.
When I am sending messages, kill PID does not work at all. I need to use kill -9 PID.
To be precise, I had made shutdown hook and then I see log "Got shutdown request ..." and
nothing more.
If I will send single message and then wait quite a long time (more than 30 or 60 sec) then
it is probable kill PID will work.
When I am sending steady couple like 2-5 messages per second mostly (>90%) kill PID does
not work - I need to use kill -9 PID.
When I am sending like 1000 or more messages per second, then kill PID does not work at all
- again kill -9 PID was doing the job.
But I suppose kill -9 PID is not the way to go, or?

Core of my shutdown hook is below. The small comment - it is not important if I have System.exit(0)
or not, as the procedure never comes to this point :-).

        Runtime.getRuntime().addShutdownHook( new Thread( () -> {
            LOG.info( "Got shutdown requests ..." );
            synchronized ( this ) {
                streams.close( 5L, TimeUnit.SECONDS );
            }

            LOG.info( "Streamer is stopped." );
            System.exit( 0 );
        }
        ) );


> KStream close( withTimeout ) - does not work under load conditions in the multi-threaded
KStream application
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5861
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5861
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.11.0.0
>            Reporter: Seweryn Habdank-Wojewodzki
>
> Recently implemented close( withTimeout ) for streams does not work under load conditions
in the multi-threaded KStream application.
> Where there are more consuming threads and there many messages in stream, then close
( withTimeout ) does not work. 
> 1. Timeout is not respected at all and
> 2. application is hanging in some streaming chaos. Theoretically threads are working
- they are busy with themselves, so the app cannot end, but they are not processing any further
messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message