hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kleegrewe, Christian" <christian.kleegr...@siemens.com>
Subject Streaming data processing and hBase
Date Fri, 16 Mar 2012 15:21:17 GMT
Dear all,

We are currently working on an architecture for a system that should be serve as an archive
for 1000+ measuring components that frequently (~30/s) send messages containing measurement
values (~300 bytes/message). The archiving system should be capable of not only serving as
a long term storage but also as a kind of streaming data processing and caching component.
There are several functions that should be computed on the incoming data before finally storing
it.

We suggested an architecture that comprises of:
A message routing component that could route data to calculations and route calculation results
to other components that are interested in these data.
An in memory caching component that is used for storing up to 10 - 20 minutes of data before
it is written to the long term archive.
An hBase database that is used for the long term storage.
MapReduce framework for doing analytics on the data stored in the hBase database.

The complete system should be failsafe and reliable regarding component failures and it should
scale with the number of computers that are utilized. 

Are there any suggestions or feedback to this approach from the community? and are there any
suggestions which tools or systems to use for the message routing component and the in memory
cache.

Thanks for any help and suggestions

all the best

Christian

----------------------------8<-----------------------------------------------

Siemens AG
Corporate Technology
Corporate Research and Technologies
CT T DE IT3
Otto-Hahn-Ring 6
81739 Munich, Germany
Tel.: +49 89 636-42722 
Fax: +49 89 636-41423 
mailto:christian.kleegrewe@siemens.com

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board:
Peter Loescher, Chairman, President and Chief Executive Officer; Roland Busch, Brigitte Ederer,
Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen,
Michael Suess; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin
Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

Mime
View raw message