incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravikumar Govindarajan <ravikumar.govindara...@gmail.com>
Subject Re: Auto-save of block-cache....
Date Wed, 20 May 2015 08:52:12 GMT
>
> Do you have any feel
> for performance of the cache warmer?  Meaning if a shard server fails do
> you have any metrics on performance differences before the cache warmer vs
> without?


I just quickly drafted a prototype to get your views on this. I am yet to
run unit-tests and surely don't have any stats as of now...

My gut feel is that cache-warmups will take around 1-2 hours for
non-trivial cache-sizes {60-70GB} with 2-4 MB/sec read throttle speeds.
Some apps may find no value addition with this.

May be we can make this optional via blur-config? Users observing no
improvements or even detrimental to existing code can switch this off…

--
Ravi

On Tue, May 19, 2015 at 6:38 PM, Aaron McCurry <amccurry@gmail.com> wrote:

> On Mon, May 18, 2015 at 7:56 AM, Ravikumar Govindarajan <
> ravikumar.govindarajan@gmail.com> wrote:
>
> > Aaron,
> >
> > I've re-worked the first-cut patch to make it more simpler. If you could
> go
> > through it when you find time, it'll be great…
> >
>
> I just read through the code.  I like it!  Thanks!  Do you have any feel
> for performance of the cache warmer?  Meaning if a shard server fails do
> you have any metrics on performance differences before the cache warmer vs
> without?  Thanks.
>
> Aaron
>
>
> >
> > --
> > Ravi
> >
> > On Thu, May 7, 2015 at 3:43 PM, Aaron McCurry <amccurry@gmail.com>
> wrote:
> >
> > > Thanks!  Will do!
> > >
> > > On Wednesday, May 6, 2015, Ravikumar Govindarajan <
> > > ravikumar.govindarajan@gmail.com> wrote:
> > >
> > > > Done… https://issues.apache.org/jira/browse/BLUR-431
> > > >
> > > > Please do review when you find time…
> > > >
> > > > --
> > > > Ravi
> > > >
> > > > On Wed, May 6, 2015 at 7:30 AM, Aaron McCurry <amccurry@gmail.com
> > > > <javascript:;>> wrote:
> > > >
> > > > > Hey Ravi,
> > > > >
> > > > > Attachments are removed from the mail list.  Perhaps you can create
> > an
> > > > > issue in jira (https://issues.apache.org/jira/browse/BLUR) and
> > attach
> > > > them
> > > > > there?  Thanks!
> > > > >
> > > > > Aaron
> > > > >
> > > > > On Tue, May 5, 2015 at 3:42 AM, Ravikumar Govindarajan <
> > > > > ravikumar.govindarajan@gmail.com <javascript:;>> wrote:
> > > > >
> > > > > > Aaron,
> > > > > >
> > > > > > I am attaching files for your reference, with minor changes
to
> > > > > > CacheDirectory. This is on 0.2.2…
> > > > > >
> > > > > > Changes look quite simple to plug-in to CacheDirectory &
> > > > BlurIndexWarmup.
> > > > > > Makes me quite suspicious of missing something obvious…
> > > > > >
> > > > > > Please do look at it when you find time...
> > > > > >
> > > > > > --
> > > > > > Ravi
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 30, 2015 at 8:25 PM, Aaron McCurry <
> amccurry@gmail.com
> > > > <javascript:;>>
> > > > > wrote:
> > > > > >
> > > > > >> Cool, let us know how it goes!
> > > > > >>
> > > > > >> Aaron
> > > > > >>
> > > > > >> On Thu, Apr 30, 2015 at 3:07 AM, Ravikumar Govindarajan
<
> > > > > >> ravikumar.govindarajan@gmail.com <javascript:;>>
wrote:
> > > > > >>
> > > > > >> > >
> > > > > >> > > Just so you are aware we are working to allow
Blur to move
> > > blocks
> > > > to
> > > > > >> > > datanodes where the shards are being served
> > > > > >> >
> > > > > >> >
> > > > > >> > Great. I remember that you were telling me this must
be the
> > > correct
> > > > > >> > approach for locality sensitive apps on hadoop.
> > > > > >> >
> > > > > >> > I am also trying the cache re-load on 0.2.2 itself..
Will
> update
> > > you
> > > > > >> once
> > > > > >> > it shapes up…
> > > > > >> >
> > > > > >> > --
> > > > > >> > Ravi
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message