flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Using Flink Streaming to write to multiple output files in HDFS
Date Mon, 09 Nov 2015 18:11:05 GMT
Hey Andra,

were you able to answer your questions from Aljoschas and Fabians links?

Flink's streaming file sink is quite unique (compared to Flume) because it
supports exactly-once semantics. Also, the performance compared to Storm is
probably much better, so you can save a lot of resources.


On Wed, Oct 21, 2015 at 2:35 PM, Fabian Hueske <fhueske@gmail.com> wrote:

> There are also training slides and programming exercises (incl. reference
> solutions) for the DataStream API at
>
> --> http://dataartisans.github.io/flink-training/
>
> Cheers, Fabian
>
> 2015-10-21 14:03 GMT+02:00 Aljoscha Krettek <aljoscha@apache.org>:
>
> > Hi,
> > the documentation has a guide about the Streaming API:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html
> >
> > This also contains a section about the rolling (HDFS) FileSystem sink:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#hadoop-filesystem
> >
> > For blog entries I would suggest these:
> >  -
> >
> http://data-artisans.com/real-time-stream-processing-the-next-step-for-apache-flink/
> >  -
> >
> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
> >  - http://data-artisans.com/kafka-flink-a-practical-how-to/
> >
> > I don’t think we have an easy starter issues right now on the Streaming
> > API. But some might come up in the future. :D
> >
> > Cheers,
> > Aljoscha
> > > On 21 Oct 2015, at 11:40, Andra Lungu <lungu.andra@gmail.com> wrote:
> > >
> > > Hey guys,
> > >
> > > Long time, no see :). I recently started a new job and it involves
> > > performing a set of real-time data analytics using Apache Kafka, Storm
> > > and Flume.
> > >
> > > What happens, on a very high level, is that set of signals is
> > > collected, stored into a Kafka topic and then Storm is used to filter
> > > certain fields out or to enrich the fields with other
> > > meta-information. Finally, Flume writes the output into mutiple HDFS
> > > files depending on the date, hour etc.
> > >
> > > Now, I saw that Flink can play with a similar pipeline, but without
> > > needing Flume for the writing to HDFS part (see
> > > http://data-artisans.com/kafka-flink-a-practical-how-to/). Which
> > > brings me to my question: jow does Flink handle writing to multiple
> > > files in a streaming fashion? -until now, I was playing with batch and
> > > writeAsCsv just took one file as a parameter-
> > >
> > > Next question: What are the prerequisites to deploy a Flink Streaming
> > > job on a cluster? Yarn, HDFS, anything else?
> > >
> > > Final question, more of a request: I'd like to play around with Flink
> > > Streaming to state whether it can substitute Storm in this use case
> > > and whether it can outrun it :P. To this end, I'll need some starting
> > > points: docs, blog posts, examples to read. Any input would be useful.
> > >
> > > I wanted to dig for a newbie task in the streaming area, but I could
> > > not find one... can we think of something easy to get me started?
> > >
> > > Thanks! Hope you guys had fun at Flink Forward!
> > > Andra
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message