flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Re: Terminating streaming test
Date Wed, 07 Feb 2018 01:16:42 GMT
Hi Thomas,

Normally the streaming job will terminate when the sources are exhausted and all records have
been processed.

I assume you have some unbounded source(s), thus this doesn’t work for your case.

We’d run into a similar situation with a streaming job that has iterations.

Our solution was your option #1 below, where we created a modified version of LocalStreamEnvironment
that supports async execution.

— Ken

> On Feb 6, 2018, at 4:21 PM, Thomas Weise <thw@apache.org> wrote:
> Hi,
> I'm looking for an example of an integration test that runs a streaming job
> and terminates when the expected result becomes available. I could think of
> 2 approaches:
> 1. Modified version of LocalStreamEnvironment that executes the job
> asynchronously and polls for the result or
> 2. Source that emits a final watermark that causes the topology to
> terminate after the watermark has traversed the topology. Is that possible
> with Flink?
> But probably this is a rather common testing need that's already solved?!
> Thanks,
> Thomas

Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message