flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
Date Wed, 28 Sep 2016 13:18:03 GMT
The Bahir-Flink stuff is fairly new - the first release has not yet
happened.
Robert is pushing that, will probably happen once he is back.

On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <shijinkui@huawei.com> wrote:

> It's nice. Will present flink source connector be pushed to bahir-flink?
> I can add netty-source to bahir-flink.
>
> Maven repository have no bahir-flink's.
> https://mvnrepository.com/artifact/org.apache.bahir
>
> -----邮件原件-----
> 发件人: Greg Hogan [mailto:code@greghogan.com]
> 发送时间: 2016年9月27日 20:58
> 收件人: dev@flink.apache.org
> 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
>
> Apache Bahir's website only suggests support for additional frameworks,
> but there is a Flink repository at
>   https://github.com/apache/bahir-flink
>
> On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijinkui@huawei.com> wrote:
>
> > Hey, Stephan Ewen
> >
> > 1.      bahir's target is spark. The contributer are rxin, srowen, tdas,
> > mateiz and so on.
> >         If we want bahir used by flink, we can suggest bahir provide
> > streaming connecter interface, such as store(), start(), stop(),
> > restart(), receiving(Any)...
> >         Then same streaming connector can be implemented by spark and
> > flink. But I think this is impossible, as bahir depend spark-streaming
> > and spark sql.
> > 2.      About connector fault tolerance. Bahir's mqtt and akka connector
> > are themselves' storage. But netty have no persist data feature.
> >         I think we can append data to a ringbuffer. When SourceContext
> > collect() throw error, then write message to inform client to stop
> > send message. When flink SourceContext is normal, then write the
> > ringbuffer data to flink, inform client to go on.
> >         Because pushing mode is hard to control the flow throughput,
> > the upstream client can.
> >         This netty connector's purpose is end-to-end streaming,
> > minimum time delay.
> > 3.      Later on, We can provide http protocol ability, put, post, get,
> > even websocket or jersey restful based on netty.
> >
> > -----邮件原件-----
> > 发件人: Stephan Ewen [mailto:sewen@apache.org]
> > 发送时间: 2016年9月27日 16:54
> > 收件人: dev@flink.apache.org
> > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> >
> > I think that could be an interesting source. Two quick questions to
> > move forward
> >
> >   - To keep the Flink code base from becoming too big (hard to
> > maintain and
> > test) we started working with Apache Bahir as a project dedicated to
> > streaming connectors. Would that be a good target for the connector?
> >
> >   - What are your thoughts on fault tolerance for that connector?
> >
> > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijinkui@huawei.com> wrote:
> >
> > > Hi, all
> > >
> > > 1.    In order to support end-to-end pushed source, I create
> FLINK-4630<
> > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > whether is this idea worth?
> > >
> > > -------------------
> > > When source stream get start, listen a provided tcp port, receive
> > > stream data from user data source.
> > > This netty tcp source is keepping alive and end-to-end, that is from
> > > business system to flink worker directly.
> > >
> > > user app push ----->  netty server source of Flink
> > >
> > > describe the source in detail below:
> > >
> > > 1.    source run as a netty tcp server
> > > 2.    user provide a tcp port, if the port is in used, increace the
> port
> > > number between 1024 to 65535. Source can parallel.
> > > 3.    callback the provided url to report the real port to listen
> > > 4.    user push streaming data to netty server, then collect the data
> to
> > > flink
> > >
> > >
> > > Thanks
> > >
> > > Jinkui Shi
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message