flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Executing detached data stream programs
Date Mon, 30 May 2016 19:30:20 GMT
Hi Jordan,

the ./bin/flink client's run command has a -d / --detached flag for
detached execution.
However, this doesn't allow you to programatically control the running job.
What you probably have to do is using the RemoteEnvironment submitting the
job in a blocking way using a separate thread.
Then, another thread can monitor the application's progress (using the
jobmanager's REST APIs) or some custom monitoring.

I know that this is not very handy right now, and there are more arguments
for having a nice API for other features like metrics, accumulators, state
queries or data samples. But I think such features will be added to Flink

On Mon, May 30, 2016 at 4:14 PM, Jordan Ganoff <jordan@ganoff.com> wrote:

> Hi,
> I plan to leveraging Flink data stream programs within a larger
> application. I’d like to be able to execute a data stream program in
> detached mode directly from the StreamExecutionEnvironment similar to how I
> can execute a program in blocking mode. I was expecting to find
> StreamExecutionEnvironment.executeDetached(). What’s the best practice for
> executing a detached data stream program and monitoring it’s progress?
> From what I can tell one approach could be to refactor the client
> configuration and interaction logic from CliFrontend into a more
> embed-friendly API.
> Thoughts?
> Thanks,
> Jordan Ganoff

View raw message