flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: 答复: 答复: [DISCUSS] add netty tcp/restful pushed source support
Date Wed, 12 Oct 2016 14:07:11 GMT
+1 to add the source to Bahir

That is an easier way to iterate fast on this and release quickly

On Wed, Oct 12, 2016 at 10:38 AM, Robert Metzger <rmetzger@apache.org>
wrote:

> Just a quick update on this one: The bahir community started already
> discussing the first bahir-flink release. I expect it to happen soon.
>
> I would really like to see the netty source in Bahir.
>
> On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen <sewen@apache.org> wrote:
>
> > The Bahir-Flink stuff is fairly new - the first release has not yet
> > happened.
> > Robert is pushing that, will probably happen once he is back.
> >
> > On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <shijinkui@huawei.com> wrote:
> >
> > > It's nice. Will present flink source connector be pushed to
> bahir-flink?
> > > I can add netty-source to bahir-flink.
> > >
> > > Maven repository have no bahir-flink's.
> > > https://mvnrepository.com/artifact/org.apache.bahir
> > >
> > > -----邮件原件-----
> > > 发件人: Greg Hogan [mailto:code@greghogan.com]
> > > 发送时间: 2016年9月27日 20:58
> > > 收件人: dev@flink.apache.org
> > > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support
> > >
> > > Apache Bahir's website only suggests support for additional frameworks,
> > > but there is a Flink repository at
> > >   https://github.com/apache/bahir-flink
> > >
> > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijinkui@huawei.com>
> wrote:
> > >
> > > > Hey, Stephan Ewen
> > > >
> > > > 1.      bahir's target is spark. The contributer are rxin, srowen,
> > tdas,
> > > > mateiz and so on.
> > > >         If we want bahir used by flink, we can suggest bahir provide
> > > > streaming connecter interface, such as store(), start(), stop(),
> > > > restart(), receiving(Any)...
> > > >         Then same streaming connector can be implemented by spark and
> > > > flink. But I think this is impossible, as bahir depend
> spark-streaming
> > > > and spark sql.
> > > > 2.      About connector fault tolerance. Bahir's mqtt and akka
> > connector
> > > > are themselves' storage. But netty have no persist data feature.
> > > >         I think we can append data to a ringbuffer. When
> SourceContext
> > > > collect() throw error, then write message to inform client to stop
> > > > send message. When flink SourceContext is normal, then write the
> > > > ringbuffer data to flink, inform client to go on.
> > > >         Because pushing mode is hard to control the flow throughput,
> > > > the upstream client can.
> > > >         This netty connector's purpose is end-to-end streaming,
> > > > minimum time delay.
> > > > 3.      Later on, We can provide http protocol ability, put, post,
> get,
> > > > even websocket or jersey restful based on netty.
> > > >
> > > > -----邮件原件-----
> > > > 发件人: Stephan Ewen [mailto:sewen@apache.org]
> > > > 发送时间: 2016年9月27日 16:54
> > > > 收件人: dev@flink.apache.org
> > > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
> > > >
> > > > I think that could be an interesting source. Two quick questions to
> > > > move forward
> > > >
> > > >   - To keep the Flink code base from becoming too big (hard to
> > > > maintain and
> > > > test) we started working with Apache Bahir as a project dedicated to
> > > > streaming connectors. Would that be a good target for the connector?
> > > >
> > > >   - What are your thoughts on fault tolerance for that connector?
> > > >
> > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijinkui@huawei.com>
> > wrote:
> > > >
> > > > > Hi, all
> > > > >
> > > > > 1.    In order to support end-to-end pushed source, I create
> > > FLINK-4630<
> > > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know
> > > > > whether is this idea worth?
> > > > >
> > > > > -------------------
> > > > > When source stream get start, listen a provided tcp port, receive
> > > > > stream data from user data source.
> > > > > This netty tcp source is keepping alive and end-to-end, that is
> from
> > > > > business system to flink worker directly.
> > > > >
> > > > > user app push ----->  netty server source of Flink
> > > > >
> > > > > describe the source in detail below:
> > > > >
> > > > > 1.    source run as a netty tcp server
> > > > > 2.    user provide a tcp port, if the port is in used, increace the
> > > port
> > > > > number between 1024 to 65535. Source can parallel.
> > > > > 3.    callback the provided url to report the real port to listen
> > > > > 4.    user push streaming data to netty server, then collect the
> data
> > > to
> > > > > flink
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Jinkui Shi
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message