hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: 2 bucket caches?
Date Mon, 29 Jun 2015 15:19:49 GMT
Hi Michael,

All what you said is exactly what I have in mind. Being able to have a
layered hierarchy with different storage engines ((Flash, SSD, Memory,
etc.) that you can configure. It's not just offheap, but anything which can
offload the BlockCache and which is still faster than going to the spinning
drives.

JM

2015-06-29 10:43 GMT-04:00 Michael Segel <michael_segel@hotmail.com>:

> I think you may want to think a bit about this…
>
> How far do you want to go with your memory management?
>
> 'Off heap' is a new nifty way of saying application level swap and memory
> management.  So what you are basically saying is that I have memory, local
> persistence, then HDFS persistence.
> And your local persistence could be anything… (PCIe based flash,
> UltraDIMMs, RRAM (when it hits the market), SSDs, even raided spinning
> rust… )
>
> If you’re going in that direction, what is tachyon doing?
>
> If you want to do this… and I’m not saying its a bad idea, you’ll want to
> think a bit more generic. Essentially its a layered hiearchy (memory, p1,
> p2, …) where p(n) is a pool of devices which have a set of rules on how to
> propagate pages up or down the hierarchy.
>
>
>
>
>
> > On Jun 29, 2015, at 1:20 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
> >
> > Hi,
> >
> > Is it possible to have 2 bucket cache on a single region server?
> >
> > Like L2 and L3? I would like to have L2 offheap and block evicted from L2
> > going into L3 on SSD. So we already have something like that? Or should I
> > open a JIRA?
> >
> > hbase.bucketcache.ioengine can get only one value. Might be nice to have
> a
> > flume-like approach...
> >
> > hbase.bucketcache=myoffheap,myssddrive
> > hbase.bucketcache.myoffheap.ioengine=offheap
> > hbase.bucketcache.myssddrive.ioengine=file://my_ssd_mnt/there
> >
> > And keep the order specified in hbase.bucketcache, so myoffheap=L2,
> > myssddrive=L3, etc.?
> >
> > Thanks,
> >
> > JM
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message