Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 74.125.82.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=ngDVQDENyod1ncwwxIHG4tjfAF2xIIFJuPBgugG2melBqcF0thNgs5w2UzOfDi+9jo
         Gfq6bVeWKocdLDXXto9kSSCOGCg0Jh3DItO9jN40YD/CEi+PnqbLJ08/KIZtrbIiNv+h
         IcXG5CD6N/58KPksTZnsd3+RWPOcW6ezvdXKg=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <AANLkTimUuK=wBFLPQE7sFAfkbXANsnBHuUFmC7X-ew67@mail.gmail.com>
References: <AANLkTimUuK=wBFLPQE7sFAfkbXANsnBHuUFmC7X-ew67@mail.gmail.com>
Date: Fri, 1 Oct 2010 08:31:02 -0700
Message-ID: <AANLkTinx_+93WF-FVLQ25fpcALOHTTsKSVsRm=Cb97ou@mail.gmail.com>
Subject: Re: memstore to blockcache
From: Stack <stack@duboce.net>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Not currently.

Blockcache is made of blocks pulled from HDFS.  It'd be a little
awkward inserting hot records into the block cache w/o going via HDFS.

But, yes, you have a good point that flush is disruptive of hot records.

In the past we talked of a keyvalue cache on top of the block cache
but it fell out of favor because block cache seemed to be good enough
but sounds like we need to revive it or do some fancy dancing if
column family is marked in-memory, we keep around the snapshot of
memstore until we know the block cache has been populated?

Any other suggestions?

File an issue.  We need to come up w/ a fix for this case.

Thanks for writing the list,
St.Ack


On Fri, Oct 1, 2010 at 1:26 AM, Abhijit Pol <apol@rocketfuel.com> wrote:
> we are trying to read efficiently a hot column family (in_memory=true,
> blockcaching=true) that get writes at say 500 qps and reads at 10,000 qps.
>
> - as long as writes are in memstore we get them from memstore and its fast
> - if we have read it once it will be at least in block cache (gets priority
> due to in_memory=true) and subsequent reads are faster
> - however memstore flush puts records on disk which demands for disk IO to
> get them back in block cache
>
> is there a way for memstore flush to go to blockcache?
>
>
> -- Abhijit
>