flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Kreutzfeldt <mnx...@gmail.com>
Subject Re: On integrating Flink with Apache NiFi
Date Tue, 22 Sep 2015 09:17:07 GMT
Hi Slim,

thanks for sharing the presentation. Like Flavio I see the (UI)
capabilities provided by Apache NiFi as very helpful for (enterprise)
customers.

At the Otto Group we are currently think about how to track data through a
stream based system like we do in the batch world using staging layers.
Apache NiFi seems to provide a feasible approach which gives lots of
insights into your running system. Especially in those cases where
topologies are still under development or people are curious on how these
mysterious streaming systems work UI supported development & tracking /
monitoring is a big win.

Having these requirements in mind, I don't see an integration point with
Apache NiFi as the project does not only provide an UI/orchestration layer
but has a runtime etc. available too. An integration like Spark (shown
right at the beginning of the demo) seems to be possible but what I'd like
to see is a more Apache NiFi-like UI layer exclusively provided for Apache
Flink. It would bring Flink very (!!!) much closer to the enterprise market
as it solves some crucial requirements when it comes to development and
monitoring - no need for developers with in-depth programming knowledge
since topologies may be built visually, no need for depth tech
understanding when it comes to monitoring ...

>From my point of view, solving the issue includes the following aspects:

* providing an appropriate UI layer to visualize data flow / operator
graphs along with metrics (system & user-defined) exported by operators
* message tracking by integrating K/V stores and index management tools

...probably both worth more than a year development time ;-)

But if there were some efforts to implement & integrate such a feature,
text me if dev support is require.

Best regards,
  Christian


2015-09-19 12:42 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:

> As a user that would be very helpful!
> On 19 Sep 2015 12:34, "Slim Baltagi" <sbaltagi@gmail.com> wrote:
>
> > Hi Flink experts!
> >
> > I came across Apache Nifi https://nifi.apache.org
> > https://www.youtube.com/watch?v=sQCgtCoZyFQ
> > In the Nifi project, there is an open JIRA
> > issue:https://issues.apache.org/jira/browse/NIFI-823 to evaluate /
> provide
> > integration with Apache Flink.
> >
> > Is the integration of Flink with NiFi on the roadmap of the Apache Flink
> > project?
> >
> > Expected advantages for Flink when integrating with Nifi would be:
> > - GUI: Web-based user interface to design data flows
> > - Dynamic dataflows:  Modify dataflow at runtime
> > - Security: Authentication, authorization, encryption
> > - Data Provenance: Track data flow from beginning to end
> >
> > What do you think?
> >
> > Thanks
> >
> > Slim Baltagi
> >
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-integrating-Flink-with-Apache-NiFi-tp8059.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message