hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N Keywal <nkey...@gmail.com>
Subject Re: Streaming data processing and hBase
Date Fri, 16 Mar 2012 17:01:53 GMT
Hi,

The way you describe the "in memory caching component", it looks very
similar to HBase memstore. Any reason for not relying on it?

N.

On Fri, Mar 16, 2012 at 4:21 PM, Kleegrewe, Christian <
christian.kleegrewe@siemens.com> wrote:

> Dear all,
>
> We are currently working on an architecture for a system that should be
> serve as an archive for 1000+ measuring components that frequently (~30/s)
> send messages containing measurement values (~300 bytes/message). The
> archiving system should be capable of not only serving as a long term
> storage but also as a kind of streaming data processing and caching
> component. There are several functions that should be computed on the
> incoming data before finally storing it.
>
> We suggested an architecture that comprises of:
> A message routing component that could route data to calculations and
> route calculation results to other components that are interested in these
> data.
> An in memory caching component that is used for storing up to 10 - 20
> minutes of data before it is written to the long term archive.
> An hBase database that is used for the long term storage.
> MapReduce framework for doing analytics on the data stored in the hBase
> database.
>
> The complete system should be failsafe and reliable regarding component
> failures and it should scale with the number of computers that are utilized.
>
> Are there any suggestions or feedback to this approach from the community?
> and are there any suggestions which tools or systems to use for the message
> routing component and the in memory cache.
>
> Thanks for any help and suggestions
>
> all the best
>
> Christian
>
>
> ----------------------------8<-----------------------------------------------
>
> Siemens AG
> Corporate Technology
> Corporate Research and Technologies
> CT T DE IT3
> Otto-Hahn-Ring 6
> 81739 Munich, Germany
> Tel.: +49 89 636-42722
> Fax: +49 89 636-41423
> mailto:christian.kleegrewe@siemens.com
>
> Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard
> Cromme; Managing Board: Peter Loescher, Chairman, President and Chief
> Executive Officer; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe
> Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y.
> Solmssen, Michael Suess; Registered offices: Berlin and Munich, Germany;
> Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684;
> WEEE-Reg.-No. DE 23691322
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message