hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Architecture question.
Date Wed, 24 Dec 2008 10:45:12 GMT
aakash shah wrote:
> We can assume that this record has only one key->value mapping. Value will be updated
every minute. Currently we have 1 Million these ( key->value ) pairs but I have to make
sure that we can scale it upto 10 million of these ( key-> value ) pairs.
> 
>   Every 10 minute I will be updating all of these value using their keys. This is the
reason I cannot go for database as a solution. 

I wouldn't be so quick to dismiss a database.  All your big telcos run 
their mobile phone systems on databases, where the big issue is having 
enough memory for the DB to stay in memory; some dedicated databases 
(e.g. TimesTen) are designed to have bounded latency on lookup so you 
can predict how long operations will take.

That said, if you are only doing atomic updates of a single record, 
there's less need for the advanced features. Assuming >1 machine, some 
kind of distributed hash table may work


>    I was thinking about going with memcache pool. In the mean-time I heard about hadoop
and wanted to get advice from this mailing list regarding memcache pool vs hadoop for this
specific problem.

It's not an area Hadoop deals with at all.

Mime
View raw message