beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <>
Subject [jira] [Commented] (BEAM-153) Support timeout in runner API
Date Wed, 30 Mar 2016 17:52:25 GMT


Davor Bonaci commented on BEAM-153:

I think it makes sense for streaming pipelines too.

This certainly can be generalized, but it would be easier to make it runner-specific for now.
Service is the one who can do TTL on a job -- hacking around BlockingDataflowPipelineRunner
is not a particularly great solution.

> Support timeout in runner API
> -----------------------------
>                 Key: BEAM-153
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-dataflow
>            Reporter: Eugene Kirpichov
> Some users want to make sure that their pipeline doesn't run longer than X minutes (e.g.
because sometimes it runs longer than that due to bugs, and in that case they'd rather auto-cancel
it than incur the costs).
> The runner API should have a timeout option, so that if the pipeline isn't in a terminal
state by then, it is automatically cancelled.
> Naturally, this only applies to batch pipelines.
> A simple way to implement this for a blocking runner (such as BlockingDataflowPipelineRunner)
is a wrapper of the sort "start pipeline, and cancel it after timeout" inside run(). For a
non-blocking runner this will require support on the underlying execution environment side.

This message was sent by Atlassian JIRA

View raw message