flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Re: Processing millions of messages in milliseconds real time -- Architecture guide required
Date Thu, 21 Apr 2016 13:29:39 GMT
This seems pretty similar to what you’re asking about:

http://data-artisans.com/extending-the-yahoo-streaming-benchmark/

Especially the part where they “...directly exposed the in-flight windows to be queried”,
as that sounds like what you meant by “The data cache should have the capability to serve
the historical data in milliseconds”

— Ken



> On Apr 18, 2016, at 10:03pm, Deepak Sharma <deepakmca05@gmail.com> wrote:
> 
> Hi all,
> I am looking for an architecture to ingest 10 mils of messages in the real time streaming
mode.
> If anyone has worked on similar kind of architecture  , can you please point me to any
documentation around the same like what should be the architecture , which all components/big
data ecosystem tools should i consider etc.
> The messages has to be in xml/json format , a preprocessor engine or message enhancer
and then finally a processor.
> I thought about using data cache as well for serving the data 
> The data cache should have the capability to serve the historical  data in milliseconds
(may be upto 30 days of data)
> 
> 
> -- 
> Thanks
> Deepak

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr




Mime
View raw message