hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kleegrewe, Christian" <christian.kleegr...@siemens.com>
Subject AW: Streaming data processing and hBase
Date Fri, 16 Mar 2012 18:39:57 GMT
Hi

Is this memstore replicated? Since we store a significant amount of data in the memory cache
we need a replicated solution. Also I can't find lots of information besides a java api doc
for the MemStore class. I will continue searching for this, but if you have any URL with more
documentation please send it. Thanks in advance

regards

Christian

--------------------------------------------8<--------------------------------------
Siemens AG 
Corporate Technology
Corporate Research and Technologies 
CT T DE IT3 
Otto-Hahn-Ring 6 
81739 München, Deutschland 
Tel.: +49 89 636-42722 
Fax: +49 89 636-41423 
mailto:christian.kleegrewe@siemens.com 

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter
Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara
Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft:
Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München,
HRB 6684; WEEE-Reg.-Nr. DE 23691322 


-----Ursprüngliche Nachricht-----
Von: N Keywal [mailto:nkeywal@gmail.com] 
Gesendet: Freitag, 16. März 2012 18:02
An: user@hbase.apache.org
Betreff: Re: Streaming data processing and hBase

Hi,

The way you describe the "in memory caching component", it looks very
similar to HBase memstore. Any reason for not relying on it?

N.

On Fri, Mar 16, 2012 at 4:21 PM, Kleegrewe, Christian <
christian.kleegrewe@siemens.com> wrote:

> Dear all,
>
> We are currently working on an architecture for a system that should be
> serve as an archive for 1000+ measuring components that frequently (~30/s)
> send messages containing measurement values (~300 bytes/message). The
> archiving system should be capable of not only serving as a long term
> storage but also as a kind of streaming data processing and caching
> component. There are several functions that should be computed on the
> incoming data before finally storing it.
>
> We suggested an architecture that comprises of:
> A message routing component that could route data to calculations and
> route calculation results to other components that are interested in these
> data.
> An in memory caching component that is used for storing up to 10 - 20
> minutes of data before it is written to the long term archive.
> An hBase database that is used for the long term storage.
> MapReduce framework for doing analytics on the data stored in the hBase
> database.
>
> The complete system should be failsafe and reliable regarding component
> failures and it should scale with the number of computers that are utilized.
>
> Are there any suggestions or feedback to this approach from the community?
> and are there any suggestions which tools or systems to use for the message
> routing component and the in memory cache.
>
> Thanks for any help and suggestions
>
> all the best
>
> Christian
>
>
> ----------------------------8<-----------------------------------------------
>
> Siemens AG
> Corporate Technology
> Corporate Research and Technologies
> CT T DE IT3
> Otto-Hahn-Ring 6
> 81739 Munich, Germany
> Tel.: +49 89 636-42722
> Fax: +49 89 636-41423
> mailto:christian.kleegrewe@siemens.com
>
> Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard
> Cromme; Managing Board: Peter Loescher, Chairman, President and Chief
> Executive Officer; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe
> Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y.
> Solmssen, Michael Suess; Registered offices: Berlin and Munich, Germany;
> Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684;
> WEEE-Reg.-No. DE 23691322
>

Mime
View raw message