hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Serving from memory
Date Tue, 01 Apr 2008 16:41:07 GMT
Chatting on list, was thought that an InMemoryMapFile would not be too 
hard to do.. couple of days, maybe.  In HBaseMapFile, could read all of 
the data into memory, into arrays (since it already sorted) as we do now 
reading in the index.  The HBaseMapFile.Reader would be modified to go 
get entries from in-memory rather than from disk.

Would be ugly since flags would have to go down through multiple levels 
of inheritance -- down through BloomFilterMapFile, HalfMapFile -- and 
that it should get cleaned up when we do our own Mapfile.

But was thought, without more intelligent rebalancing of regions over 
cluster so they were evenly distributed, with our current lumpy 
assignment, it would be easy for a regionserver to have its memory 
overstrained; it'd go down with an OOME, regions would be redistributed 
lumpy, another would go down and then a downward spiral.  Was thought 
this had to be addressed first.

(That a fair summary Bryan?)


Bryan Duxbury wrote:
> I'm not thinking of hot cell caching. I'm talking about going the 
> whole way and putting all the data in-memory. So yes, store file 
> contents would be loaded into memory, though not the memcache, because 
> that would get really complicated, I think. InMemoryStoreFile would 
> really be what I was going for, I'd guess. The table wouldn't be 
> read-only. Writes would go through to disk but reads would come 
> straight from memory.
> On Mar 31, 2008, at 8:42 PM, stack wrote:
>> A Reference-cache of hot cells would take a day at the outside I'd 
>> guess.  The bulk of the work is done.
>> If you're talking about something else, lets discuss.  What would it 
>> look like?  Store MapFiles would be floated in memory or copied to 
>> MemCache?  We'd need a special In-Memory MapFile?  We'd do a bulk 
>> memcopy from HDFS up into mem and then you'd serve from there?  Would 
>> the table have to be read-only?
>> St.Ack
>> Bryan Duxbury wrote:
>>> Quick poll for us devs - if you had to guess, how long do you think 
>>> it would take for the in-memory option of HBase to actually be 
>>> implemented to work reliably?
>>> -Bryan

View raw message