flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shijinkui <shijin...@huawei.com>
Subject 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
Date Tue, 27 Sep 2016 13:31:08 GMT
It's nice. Will present flink source connector be pushed to bahir-flink?
I can add netty-source to bahir-flink.

Maven repository have no bahir-flink's.
https://mvnrepository.com/artifact/org.apache.bahir

-----邮件原件-----
发件人: Greg Hogan [mailto:code@greghogan.com] 
发送时间: 2016年9月27日 20:58
收件人: dev@flink.apache.org
主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support

Apache Bahir's website only suggests support for additional frameworks, but there is a Flink
repository at
  https://github.com/apache/bahir-flink

On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijinkui@huawei.com> wrote:

> Hey, Stephan Ewen
>
> 1.      bahir's target is spark. The contributer are rxin, srowen, tdas,
> mateiz and so on.
>         If we want bahir used by flink, we can suggest bahir provide 
> streaming connecter interface, such as store(), start(), stop(), 
> restart(), receiving(Any)...
>         Then same streaming connector can be implemented by spark and 
> flink. But I think this is impossible, as bahir depend spark-streaming 
> and spark sql.
> 2.      About connector fault tolerance. Bahir's mqtt and akka connector
> are themselves' storage. But netty have no persist data feature.
>         I think we can append data to a ringbuffer. When SourceContext
> collect() throw error, then write message to inform client to stop 
> send message. When flink SourceContext is normal, then write the 
> ringbuffer data to flink, inform client to go on.
>         Because pushing mode is hard to control the flow throughput, 
> the upstream client can.
>         This netty connector's purpose is end-to-end streaming, 
> minimum time delay.
> 3.      Later on, We can provide http protocol ability, put, post, get,
> even websocket or jersey restful based on netty.
>
> -----邮件原件-----
> 发件人: Stephan Ewen [mailto:sewen@apache.org]
> 发送时间: 2016年9月27日 16:54
> 收件人: dev@flink.apache.org
> 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
>
> I think that could be an interesting source. Two quick questions to 
> move forward
>
>   - To keep the Flink code base from becoming too big (hard to 
> maintain and
> test) we started working with Apache Bahir as a project dedicated to 
> streaming connectors. Would that be a good target for the connector?
>
>   - What are your thoughts on fault tolerance for that connector?
>
> On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijinkui@huawei.com> wrote:
>
> > Hi, all
> >
> > 1.    In order to support end-to-end pushed source, I create FLINK-4630<
> > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know 
> > whether is this idea worth?
> >
> > -------------------
> > When source stream get start, listen a provided tcp port, receive 
> > stream data from user data source.
> > This netty tcp source is keepping alive and end-to-end, that is from 
> > business system to flink worker directly.
> >
> > user app push ----->  netty server source of Flink
> >
> > describe the source in detail below:
> >
> > 1.    source run as a netty tcp server
> > 2.    user provide a tcp port, if the port is in used, increace the port
> > number between 1024 to 65535. Source can parallel.
> > 3.    callback the provided url to report the real port to listen
> > 4.    user push streaming data to netty server, then collect the data to
> > flink
> >
> >
> > Thanks
> >
> > Jinkui Shi
> >
> >
>
Mime
View raw message