hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Re: performance of block cache
Date Wed, 20 Aug 2014 17:48:09 GMT
Hi Zhaojie,

I'm responsible for this particular bit of work. One thing to note in these
experiments is that I did not control explicitly for OS caching. I ran
"warmup" workloads before collecting measurements, but because the amount
of RAM on the machine is fixed, it's impact of OS cache is different with
different based on the amount of memory used by HBase. Another, as Todd
pointed out on an earlier thread, is that my trend lines are probably
optimistic/misleading.

Something I was driving for was to understand how well the different
implementations before as they're managing more and more memory. I'd like
to get some insight into how we might be able to take advantage of 100's or
even 1000's of GB of memory when the time comes. That's part of why there's
so many variables.

I scripted out the running of the tests, all of my configurations are
available in the associated github repo [0], and all of the data points are
available as a csv. If you're interested in experimenting yourself, please
let me know how I can help.

Cheers,
Nick

[0]: https://github.com/ndimiduk/perf_blockcache


On Wed, Aug 20, 2014 at 6:00 AM, 牛兆捷 <nzjemail@gmail.com> wrote:

> the complete blog link is:
> http://zh.hortonworks.com/blog/blockcache-showdown-hbase/
>
>
> 2014-08-20 11:41 GMT+08:00 牛兆捷 <nzjemail@gmail.com>:
>
> > Hi all:
> >
> > I saw some interesting results from Hortonworks blog (block cache
> > <
> http://zh.hortonworks.com/wp-content/uploads/2014/03/perfeval_blockcache_v2.pdf
> >
> > ).
> >
> > In this result, the ratio of memory footprint to database size is held
> > fixed while
> > the absolute values are increased.
> >
> > In my mind, the performance should becomes worse for larger ratio as the
> > increase
> > of the absolute value. For example BucketCache#(tmpfs), the difference
> > between ratio (DB"1.5":"RAM"1.0) and ratio (DB"4.5":"RAM"1.0) becomes
> > larger as the increase of memory.
> > Actually, the result of ratio ( DB"1.5":"RAM"1.0) increase linearly, and
> > the result of ratio (DB"1.5":"RAM"1.0) exponentially.
> >
> > However, for BucketCache#(heap) and LruBlockCache, the result is out of
> my
> > expectation.
> > The curves of ratio (DB"1.5":"RAM"1.0) and ratio (DB"4.5":"RAM"1.0) both
> > increase exponentially, but the relative differences as the increase of
> > memory are not consistent.
> > Take LruBlockCache as an example, the difference of ratio
> > (DB"1.5":"RAM"1.0) and ratio (DB"4.5":"RAM"1.0) becomes smaller from 20
> GB
> > to 50 GB, but becomes larger from 50 GB to 60 GB.
> >
> > How to analysis the cause of this result, any ideas?
> >
> > --
> > *Regards,*
> > *Zhaojie*
> >
> >
>
>
> --
> *Regards,*
> *Zhaojie*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message