flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bjlovegithub <...@git.apache.org>
Subject [GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Date Fri, 04 Nov 2016 10:33:45 GMT
Github user bjlovegithub commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2629#discussion_r86520537
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/api/StreamingOperatorsITCase.java
---
    @@ -195,6 +202,70 @@ public Integer map(NonSerializable value) throws Exception {
     		env.execute();
     	}
     
    +	@Test
    +	public void testAsyncWaitOperator() throws Exception {
    +		final int numElements = 10;
    +
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +
    +		DataStream<Tuple2<Integer, NonSerializable>> input = env.addSource(new
NonSerializableTupleSource(numElements)).setParallelism(1);
    +
    +		AsyncFunction<Tuple2<Integer, NonSerializable>, Integer> function = new
RichAsyncFunction<Tuple2<Integer, NonSerializable>, Integer>() {
    +			transient ExecutorService executorService;
    +
    +			@Override
    +			public void open(Configuration parameters) throws Exception {
    +				super.open(parameters);
    +				executorService = Executors.newFixedThreadPool(numElements);
    +			}
    +
    +			@Override
    +			public void close() throws Exception {
    +				super.close();
    +				executorService.shutdown();
    +			}
    +
    +			@Override
    +			public void asyncInvoke(final Tuple2<Integer, NonSerializable> input,
    +									final AsyncCollector<Tuple2<Integer, NonSerializable>, Integer>
collector) throws Exception {
    +				this.executorService.submit(new Runnable() {
    +					@Override
    +					public void run() {
    +						// wait for while to simulate async operation here
    +						int sleep = (int) (new Random().nextFloat() * 1000);
    +						try {
    +							Thread.sleep(sleep);
    +							List<Integer> ret = new ArrayList<>();
    +							ret.add(input.f0+input.f0);
    +							collector.collect(ret);
    +						}
    +						catch (InterruptedException e) {
    +							collector.collect(new ArrayList<Integer>(0));
    +						}
    +					}
    +				});
    +			}
    +		};
    +
    +		DataStream<Integer> orderedResult = AsyncDataStream.orderedWait(input, function,
2).setParallelism(1);
    +		orderedResult.writeAsText(resultPath1, FileSystem.WriteMode.OVERWRITE).setParallelism(1);
    +
    +		DataStream<Integer> unorderedResult = AsyncDataStream.unorderedWait(input, function,
2).setParallelism(1);
    +		unorderedResult.writeAsText(resultPath2, FileSystem.WriteMode.OVERWRITE);
    --- End diff --
    
    I think it is determined by `StreamingMultipleProgramsTestBase`, which requires tmp files
from test job's output to verify the result.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message