kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seweryn Habdank-Wojewodzki (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-5861) KStream close( withTimeout ) - does not work under load conditions in the multi-threaded KStream application
Date Tue, 12 Sep 2017 13:51:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162964#comment-16162964
] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5861 at 9/12/17 1:50 PM:
----------------------------------------------------------------------------

I made further investigations. Thanks to inspiration from Ted Yu, I dumped stack when java
process was blocked.
I used "jcmd PID Thread.print" command (jstack is obsolete).
>From this stack I saw that indeed System.exit() blocks complete process.
So finally I modify my shutdown hook handler to such a code, where lock is very primitive
{{Object lock = new Object()}}:

{code:java}
        final Thread mainThread = Thread.currentThread();
        Runtime.getRuntime().addShutdownHook( new Thread( () -> {
            synchronized ( lock ) {
                LOG.info( "Got shutdown requests ..." );
                streams.close( 5L, TimeUnit.SECONDS );
                LOG.info( "Streamer is stopped." );
                try {
                    // found at: https://stackoverflow.com/questions/2921945/useful-example-of-a-shutdown-hook-in-java
                    mainThread.join( 1000 );
                } catch ( InterruptedException e ) {
                    e.printStackTrace();
                }
            }
        }
        ) );
{code}

This one seems to work and I can gracefully shutdown using kill PID command.



was (Author: habdank):
I made further investigations. Thanks to inspiration from Ted Yu, I dumped stack when java
process was blocked.
I used "jcmd PID Thread.print" command (jstack is obsolete).
>From this stack I saw that indeed System.exit() blocks complete process.
So finally I modify my shutdown hook handler to such a code, where lock is very primitive
{{Object lock = new Object()}}:

{code:java}
        final Thread mainThread = Thread.currentThread();
        Runtime.getRuntime().addShutdownHook( new Thread( () -> {
            synchronized ( lock ) {
                LOG.info( "Got shutdown requests ..." );
                streams.close( 5L, TimeUnit.SECONDS );
                LOG.info( "Streamer is stopped." );
                try {
                    // found at: https://stackoverflow.com/questions/2921945/useful-example-of-a-shutdown-hook-in-java
                    mainThread.join( 1000 );
                } catch ( InterruptedException e ) {
                    e.printStackTrace();
                }
            }
        }
        ) );
{code}


> KStream close( withTimeout ) - does not work under load conditions in the multi-threaded
KStream application
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5861
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5861
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.11.0.0
>            Reporter: Seweryn Habdank-Wojewodzki
>
> Recently implemented close( withTimeout ) for streams does not work under load conditions
in the multi-threaded KStream application.
> Where there are more consuming threads and there many messages in stream, then close
( withTimeout ) does not work. 
> 1. Timeout is not respected at all and
> 2. application is hanging in some streaming chaos. Theoretically threads are working
- they are busy with themselves, so the app cannot end, but they are not processing any further
messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message