flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Visualize result of Flink job
Date Mon, 30 May 2016 12:37:49 GMT
Hi,
sounds doable. I think it should be easy to set up a first proof of concept
for this.
Let us know if you need any further assistance.

Regards,
Robert

On Mon, May 30, 2016 at 2:29 PM, Palle <palle@sport.dk> wrote:

> Hi Robert.
>
> Thank you for the answer.
>
> I am looking at a rate of max 10.000 elements / 10 seconds, so
> Elastic/Kibana is probably the way to go. I'll find a way to model it.
>
> Thanks.
>
> /Palle
>
> ----- Original meddelelse -----
>
> *Fra:* Robert Metzger <rmetzger@apache.org>
> *Til:* user@flink.apache.org <user@flink.apache.org>
> *Dato:* Man, 30. maj 2016 12:31
>
> *Emne:* Re: Visualize result of Flink job
>
> Hi Palle,
> I think there is currently no way of sending the data from a streaming
> Flink job into Zeppelin.
> What rate / amount of data do you expect to send every 10 seconds to the
> visualization tool?
> People have used Flink -> ES -> Kibana for this purpose in the past [1],
> but I think you can not send millions of records per second into ES.
> Something like 1000 - 5000 elements / second should easily work for a small
> ES setup.
> [1]
> https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana
>
> On Mon, May 30, 2016 at 10:00 AM, Palle <palle@sport.dk> wrote:
>
>> OK, I found product that seems to be what I am looking for: Apache
>> Zeppelin. I will have a look into that one. If anyone can point me to an
>> example (Git) outputting data from Flink to the Zeppelin Notebook I would
>> be happy.
>>
>>
>> ----- Original meddelelse -----
>> > Fra: Palle <palle@sport.dk>
>> > Til: user@flink.apache.org
>> > Dato: Man, 30. maj 2016 08:20
>> > Emne: Re: Visualize result of Flink job
>> >
>> > I know exactly what to visualize. As I wrote, it is the latest
>> > result of the Flink job I would like to visualize. There is no need
>> > to use elastic to find it first.
>> >
>> > The data I have is of such a  nature that  they every 10 seconds
>> > could be written into a file, meaning that the file at all times
>> > would contain the most recent results (at latest 10 seconds old). I
>> > am not interested in the history, and therefore I should think
>> > elastic is not the best fit. So my question is if anyone knows of a
>> > component (Apache or other) that can make the visualization a little
>> > nicer than just the file :-)
>> >
>> >
>> > ----- Original meddelelse -----
>> > > Fra: Kanstantsin Kamkou <kkamkou@gmail.com>
>> > > Til: user@flink.apache.org
>> > > Dato: Søn, 29. maj 2016 22:42
>> > > Emne: Re: Visualize result of Flink job
>> > >
>> > > >  I am thinking it may not be the best fit, because Elastic is by
>> > > nature a search engine that is good for trending and stuff like
>> > that
>> > > - not entire replacement of the current view.
>> > > Why u think that the elasticsearch is not the right tool? To
>> > > visualise
>> > > something u have to find what to visualise  first, right?
>> > >
>> > >
>> > > On Sun, May 29, 2016 at 8:20 PM, Palle <palle@sport.dk> wrote:
>> > > > Hi there
>> > > >
>> > > > I am using Flink to analyse a lot of incoming data. Every 10
>> > > seconds it makes sense to present the analysis so far as some form
>> > > of visualization. Every 10 seconds I therefore will replace the
>> > > current contents of the visualization/presentation with the
>> > analysis
>> > > result of the most recent 10 seconds.
>> > > >
>> > > > I was first thinking of using ElasticSearch/Kibana for this
>> > > because I know it should be easy to set up, but I am thinking it
>> > may
>> > > not be the best fit, because Elastic is by nature a search engine
>> > > that is good for trending and stuff like that - not entire
>> > > replacement of the current view. And therefore I may also
>> > experience
>> > > difficulties implementing the view in Elastic.
>> > > >
>> > > > Does anyone know of any other visualization tools that work well
>> > > with Flink? ...where it is easy to export the result of a Flink
>> > job
>> > > to a user interface (web).
>> > > >
>> > > > Thanks
>> > > > Palle
>> >
>>
>
>
>

Mime
View raw message