flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <mbala...@apache.org>
Subject Re: UDP support in Streaming API
Date Thu, 26 Mar 2015 12:45:20 GMT
Dear Janani,

Apache Kafka as a source is supported by our system, check out the
documentation for details. [1]

You can use UDP as a source if you wish, just bear in mind the standard
disadvantages of it: the possibility of losing messages and that you will
have to manually deal with the serialization of datagrams. For adding your
custom source all you need to implement is an instance of the
SourceFunction interface. [2]

[1]
http://ci.apache.org/projects/flink/flink-docs-master/streaming_guide.html#apache-kafka
[2]
http://ci.apache.org/projects/flink/flink-docs-master/streaming_guide.html#sources

Best,

Marton

On Thu, Mar 26, 2015 at 1:15 PM, Janani Chakkaradhari <
janani.cse05@gmail.com> wrote:

> Hi Stephan,
>
> Yes, you are right. I will try writing a custom data source as you
> mentioned. Also, I still need to check the possibility of our system to use
> Apache Kafka as a broker. Could it be possible for you to point out here
> the downsides of using UDP as a source for streaming data?
>
> Thanks,
> Janani
>
> On Wed, Mar 25, 2015 at 9:44 PM, Stephan Ewen <sewen@apache.org> wrote:
>
> > Hi Janani!
> >
> > Do I understand you correctly in that you want a Flink stream source that
> > receives UDP datagrams and turns them into Flink DataStream?
> >
> > Such a thing is not in there, yet. The interface to define custom data
> > sources is rather simple, though, it should be possible to add something
> > like this.
> >
> > Two things are not clear to me however:
> >
> > 1) Where are the datagrams sent to? That would need to be the point where
> > the data source runs.
> >
> > 2) You need a way of turning the datagrams (which are just bytes) into
> > records. Would that be hard-wired in your case?
> >
> > So, while probably possible, I would guess a UDP source has quite a few
> > downsides over a message queue as a source. Using something like Kafka to
> > communicate the source data is easier and better recoverable.
> >
> > Does your setup allow to put the data into Kafka and having Flink read
> the
> > streams from Kafka?
> >
> > Greetings,
> > Stephan
> >  Am 25.03.2015 19:20 schrieb "Janani Chakkaradhari" <
> > janani.cse05@gmail.com
> > >:
> >
> > > ​Hi,
> > >
> > > Does Flink's streaming api has support for
> > > ​ reading streams of data via Java UDP (DatagramSocket)? If so kindly
> > > advise me in which release of Flink I can find it.
> > >
> > > Thanks,
> > > Janani
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message