flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Re: Terminating streaming test
Date Wed, 07 Feb 2018 01:16:42 GMT
Hi Thomas,

Normally the streaming job will terminate when the sources are exhausted and all records have
been processed.

I assume you have some unbounded source(s), thus this doesn’t work for your case.

We’d run into a similar situation with a streaming job that has iterations.

Our solution was your option #1 below, where we created a modified version of LocalStreamEnvironment
<https://raw.githubusercontent.com/ScaleUnlimited/flink-crawler/master/src/main/java/org/apache/flink/streaming/api/environment/LocalStreamEnvironmentWithAsyncExecution.java>
that supports async execution.

— Ken


> On Feb 6, 2018, at 4:21 PM, Thomas Weise <thw@apache.org> wrote:
> 
> Hi,
> 
> I'm looking for an example of an integration test that runs a streaming job
> and terminates when the expected result becomes available. I could think of
> 2 approaches:
> 
> 1. Modified version of LocalStreamEnvironment that executes the job
> asynchronously and polls for the result or
> 
> 2. Source that emits a final watermark that causes the topology to
> terminate after the watermark has traversed the topology. Is that possible
> with Flink?
> 
> But probably this is a rather common testing need that's already solved?!
> 
> Thanks,
> Thomas

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message