hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 牛兆捷 <nzjem...@gmail.com>
Subject Re: performance of block cache
Date Mon, 15 Sep 2014 02:26:16 GMT
Hi, Nick

Can your perf_blockcache performance testing script can be applied to hbase
cluster directly?
If not, what kind of things should I take care?

2014-08-22 7:06 GMT+08:00 Nick Dimiduk <ndimiduk@gmail.com>:

> I'm familiar with Stack's work too, but thanks for pointing it out :)
>
> On Wed, Aug 20, 2014 at 8:19 PM, 牛兆捷 <nzjemail@gmail.com> wrote:
>
> > Hi Nick:
> >
> > Yes, I am interested in it. I will try first.
> >
> > Btw, this site (http://people.apache.org/~stack/bc/) also does the
> similar
> > performance evaluation.
> > You can have a look if you are interested in.
> >
> >
> > 2014-08-21 1:48 GMT+08:00 Nick Dimiduk <ndimiduk@gmail.com>:
> >
> > > Hi Zhaojie,
> > >
> > > I'm responsible for this particular bit of work. One thing to note in
> > these
> > > experiments is that I did not control explicitly for OS caching. I ran
> > > "warmup" workloads before collecting measurements, but because the
> amount
> > > of RAM on the machine is fixed, it's impact of OS cache is different
> with
> > > different based on the amount of memory used by HBase. Another, as Todd
> > > pointed out on an earlier thread, is that my trend lines are probably
> > > optimistic/misleading.
> > >
> > > Something I was driving for was to understand how well the different
> > > implementations before as they're managing more and more memory. I'd
> like
> > > to get some insight into how we might be able to take advantage of
> 100's
> > or
> > > even 1000's of GB of memory when the time comes. That's part of why
> > there's
> > > so many variables.
> > >
> > > I scripted out the running of the tests, all of my configurations are
> > > available in the associated github repo [0], and all of the data points
> > are
> > > available as a csv. If you're interested in experimenting yourself,
> > please
> > > let me know how I can help.
> > >
> > > Cheers,
> > > Nick
> > >
> > > [0]: https://github.com/ndimiduk/perf_blockcache
> > >
> > >
> > > On Wed, Aug 20, 2014 at 6:00 AM, 牛兆捷 <nzjemail@gmail.com> wrote:
> > >
> > > > the complete blog link is:
> > > > http://zh.hortonworks.com/blog/blockcache-showdown-hbase/
> > > >
> > > >
> > > > 2014-08-20 11:41 GMT+08:00 牛兆捷 <nzjemail@gmail.com>:
> > > >
> > > > > Hi all:
> > > > >
> > > > > I saw some interesting results from Hortonworks blog (block cache
> > > > > <
> > > >
> > >
> >
> http://zh.hortonworks.com/wp-content/uploads/2014/03/perfeval_blockcache_v2.pdf
> > > > >
> > > > > ).
> > > > >
> > > > > In this result, the ratio of memory footprint to database size is
> > held
> > > > > fixed while
> > > > > the absolute values are increased.
> > > > >
> > > > > In my mind, the performance should becomes worse for larger ratio
> as
> > > the
> > > > > increase
> > > > > of the absolute value. For example BucketCache#(tmpfs), the
> > difference
> > > > > between ratio (DB"1.5":"RAM"1.0) and ratio (DB"4.5":"RAM"1.0)
> becomes
> > > > > larger as the increase of memory.
> > > > > Actually, the result of ratio ( DB"1.5":"RAM"1.0) increase
> linearly,
> > > and
> > > > > the result of ratio (DB"1.5":"RAM"1.0) exponentially.
> > > > >
> > > > > However, for BucketCache#(heap) and LruBlockCache, the result is
> out
> > of
> > > > my
> > > > > expectation.
> > > > > The curves of ratio (DB"1.5":"RAM"1.0) and ratio (DB"4.5":"RAM"1.0)
> > > both
> > > > > increase exponentially, but the relative differences as the
> increase
> > of
> > > > > memory are not consistent.
> > > > > Take LruBlockCache as an example, the difference of ratio
> > > > > (DB"1.5":"RAM"1.0) and ratio (DB"4.5":"RAM"1.0) becomes smaller
> from
> > 20
> > > > GB
> > > > > to 50 GB, but becomes larger from 50 GB to 60 GB.
> > > > >
> > > > > How to analysis the cause of this result, any ideas?
> > > > >
> > > > > --
> > > > > *Regards,*
> > > > > *Zhaojie*
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Regards,*
> > > > *Zhaojie*
> > > >
> > >
> >
> >
> >
> > --
> > *Regards,*
> > *Zhaojie*
> >
>



-- 
*Regards,*
*Zhaojie*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message