flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2111) Add "stop" signal to cleanly shutdown streaming jobs
Date Mon, 08 Feb 2016 12:38:39 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136892#comment-15136892

ASF GitHub Bot commented on FLINK-2111:

Github user tillrohrmann commented on the pull request:

    I strongly agree with @StephanEwen not to make the distinction between streaming and batch
jobs in the runtime explicit. Even though this might only be a cosmetic change for this PR
but who knows what other people will build on top of it once it is there. I fear that we won't
be able to remove such a flag once introduced.
    Concerning whether all `SourceFunction` should have a `stop` method: I think it is not
a priori clear that all streaming sources are gracefully stoppable. Take for example the `FlinkKafkaConsumer08`.
Currently,   the stop method is implemented by calling `cancel`. However, `cancel` does not
guarantee that the sources stop gracefully. Exceptions might be thrown and the state might
be inconsistent. In fact, with the current implementation, `stop` won't stop the `FlinkKafkaConsumer08`
at all. The `cancel` call only works in combination with the `TaskCanceler` which interrupts
the `Task` thread (in this case `LegacyFetcher.run`). If the interrupt does not happen, the
`SimpleConsumerThreads` might be stuck for all eternity in the fetch loop.
    Furthermore, adding the `stop` method to the `SourceFunction` interface will be an API
breaking change and all users would have to implement `stop`, henceforth. I think a `StoppableOperator`
interface would be more modular and less restrictive because you have the freedom not to implement
a graceful stop operation.
    What do you think?

> Add "stop" signal to cleanly shutdown streaming jobs
> ----------------------------------------------------
>                 Key: FLINK-2111
>                 URL: https://issues.apache.org/jira/browse/FLINK-2111
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime, JobManager, Local Runtime, Streaming, TaskManager,
>            Reporter: Matthias J. Sax
>            Assignee: Matthias J. Sax
>            Priority: Minor
> Currently, streaming jobs can only be stopped using "cancel" command, what is a "hard"
stop with no clean shutdown.
> The new introduced "stop" signal, will only affect streaming source tasks such that the
sources can stop emitting data and shutdown cleanly, resulting in a clean shutdown of the
whole streaming job.
> This feature is a pre-requirment for https://issues.apache.org/jira/browse/FLINK-1929

This message was sent by Atlassian JIRA

View raw message