flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4391) Provide support for asynchronous operations over streams
Date Mon, 07 Nov 2016 07:35:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643392#comment-15643392

ASF GitHub Bot commented on FLINK-4391:

Github user bjlovegithub commented on the issue:

    Maybe this is a solution. We will not think about changing the type of `checkpointLock`,
which is an `Object` ans quite efficient, and not change the order of  `broadcastBarriers`
and `operator.snapshotState()`. By placing **pause** `EmitterThread` codes in `StreamTask.performCheckpoint()`,
like this:  
    `	private boolean performCheckpoint(CheckpointMetaData checkpointMetaData) throws Exception
    		LOG.debug("Starting checkpoint {} on task {}", checkpointMetaData.getCheckpointId(),
    		synchronized (lock) {
    			if (isRunning) {
    				// stop working threads first.
    				for (StreamOperator operator : operatorChain.getAllOperators()) {
    					if (operator instanceof AsyncWaitOperator) {
    				// broadcast barriers after snapshot operators' states.
    						checkpointMetaData.getCheckpointId(), checkpointMetaData.getTimestamp()
    				return true;
    			} else {
    				return false;

> Provide support for asynchronous operations over streams
> --------------------------------------------------------
>                 Key: FLINK-4391
>                 URL: https://issues.apache.org/jira/browse/FLINK-4391
>             Project: Flink
>          Issue Type: New Feature
>          Components: DataStream API
>            Reporter: Jamie Grier
>            Assignee: david.wang
> Many Flink users need to do asynchronous processing driven by data from a DataStream.
 The classic example would be joining against an external database in order to enrich a stream
with extra information.
> It would be nice to add general support for this type of operation in the Flink API.
 Ideally this could simply take the form of a new operator that manages async operations,
keeps so many of them in flight, and then emits results to downstream operators as the async
operations complete.

This message was sent by Atlassian JIRA

View raw message