flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Qin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-6085) flink as micro service
Date Wed, 29 Mar 2017 03:04:42 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945991#comment-15945991
] 

Chen Qin edited comment on FLINK-6085 at 3/29/17 3:04 AM:
----------------------------------------------------------

To unlock scenarios like blocking rpc call or async callback, I am currently thinking of way
to connecting web front directly to pipeline in normal case.  Use some kind of durable buffer
to store requests since last successful checkpoint against failure scenario.

I think what we can do at this point is to assume client will do retry after connection failure
and flink as a micro service maintain at least once semantics. So the problem simplified to
implement a web front source and feedback loop from sink to source & locate pending connection
to response.

What do you think [~till.rohrmann] [~tudandan]

 


was (Author: foxss):
To unlock scenarios like blocking rpc call or async callback, I am currently thinking of way
to get rid of using distributed queue by connecting web front directly to pipeline.

I put a bit more thoughts on this topic, exact once seems really hard to achieve through rpc
source. Same issue as using web front ingestion to distributed queue at a matter of fact.
Clients can do arbitrary retry within long time span.

I think what we can do at this point is to assume client will do retry after connection failure
and flink as a micro service maintain at least once semantics. So the problem simplified to
implement a web front source and feedback loop from sink to source & locate pending connection
to response.

What do you think [~till.rohrmann] [~tudandan]

 

> flink as micro service
> ----------------------
>
>                 Key: FLINK-6085
>                 URL: https://issues.apache.org/jira/browse/FLINK-6085
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API, JobManager
>            Reporter: Chen Qin
>            Priority: Minor
>         Attachments: Untitled document.jpg
>
>
> Track discussion around run flink as a micro service, includes but not limited to 
> - RPC (web service endpoint) source
>   as web service endpoint accept RPC call, ingest to the streaming job(only one)
> - callback mechanism 
> - task assignment should honor deployment group (web tier hosts should be isolated from
rest of task assignment)
> https://docs.google.com/document/d/1MSsTOz7xUu50dAf_8v3gsQFfJFFy9LKnULdIl26yj0o/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message