flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Sharma <deepakmc...@gmail.com>
Subject Re: Where to put live model and business logic in Hadoop/Flink BigData system
Date Fri, 06 May 2016 11:40:59 GMT
I see the flow to be as below:
LogStash->Log Stream->Flink ->Kafka->Live Model

The Live Model will again be Flink streaming data sets from Kakfa.
There you analyze the incoming stream for the certain value and once you
find this certain value , read the historical view and then do the analysis
in Flink itself.
For your java objects , i guess you can use checkpointed interface (have
not used it though yet)


On Fri, May 6, 2016 at 4:22 PM, <palle@sport.dk> wrote:

> Hi there.
> We are putting together some BigData components for handling a large
> amount of incoming data from different log files and perform some analysis
> on the data.
> All data being fed into the system will go into HDFS. We plan on using
> Logstash, Kafka and Flink for bringing data from the log files and into
> HDFS. All our data located in HDFS we will designate as our historic data
> and we will use MapReduce (probably Flink, but could also be Hadoop) to
> create some aggregate views of the historic data. These views we will
> locate probably in HBase or MongoDB.
> These views of the historic data (also called batch views in the Lambda
> Architecture if any of you are familiar with that) we will use from the
> live model in the system. The live model is also being fed with the same
> data (through Kafka) and when the live model detects a certain value in the
> incoming data, it will perform some analysis using the views in
> HBase/MongoDB of the historic data.
> Now, could anyone share some knowledge regarding where it would be
> possible to implement such a live model given the components we plan on
> using? Apart from the business logic that will perform the analysis, our
> live model will at all times also contain a java object structure of maybe
> 5-10 java collections (maps, lists) containing approx 5 mio objects.
> So, where is it possible to implement our live model? Can we do this in
> Flink? Can we do this with another component within the Hadoop Big Data
> ecosystem?
> Thanks.
> /Palle


View raw message