flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <ni...@basj.es>
Subject Re: Comparison of storm and flink
Date Sun, 24 Jan 2016 09:34:31 GMT

I don't have a perfect list available but these are some of the things to
keep in mind:
1) end2end Latency. Some systems (like spark) use microbatching which
introduces a latency of seconds
2) Do you get "exactly once guarantees"? Storm can give you that but then
the throughput goes really down.
3) Ease of programming. How 'nice' is the api you have to work with.
4) Resiliance of state. If you need some state over several events, Does
the framework support this and has built in recovery of this stat in case
of a faillure?
5) Tools. What kind of tools are " ready to run" available? I.e. kmeans,
and things like that.
6) Deployment. How do you run it? Do you need a separate infrastructure or
can you deploy it in an existing yarn/mesos/...
7) Security: Can it access kerberos secured resources (like hbase, hdfs or
any other service) in a long running situation.

As a final note: I've been hacking at Storm for over a year now and last
summer I found Flink. Today Storm is for me no longer an option and we are
taking down what we already had running.

Niels Basjes
On 23 Jan 2016 20:59, "Vinaya M S" <vinayams@gmail.com> wrote:

> Hi Flink user group,
> I am working on a project for the Insight Data Engineering Program in New
> York to compare streaming tools. The program is designed for software
> engineers and those straight from the university to transition to a data
> engineering role.  After completing the project, we present demos of the
> project to several companies in NYC that we are interested in working for
> (including top companies like NY Times, Capital One, Bloomberg, etc).
> I have decided to work on a project to compare streaming tools, namely
> Flink and Storm.  I already have Twitter data stored and would like to
> design tests to benchmark the the two tools if possible.
> I wanted to be extra-careful in constructing a benchmark to work on and
> present at companies here in NY.  Do you have any recommendations to tests
> to run with the Twitter data that I have that would showcase when to and
> not use Flink compared to Storm?
> Thanks!
> Vinaya

View raw message