hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Offheap config question for Hbase 1.1.2
Date Tue, 12 Sep 2017 17:14:33 GMT
Ram:
As Arul indicated, HDP is being used where there're a lot of backports to
1.1.2 baseline.

Arul:
I suggest going thru vendor's channel first.

Cheers

On Tue, Sep 12, 2017 at 10:11 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Why Master also accepts MaxDirectMemory config is probably because in 1.1
> onwards we treat HMaster as HregionServer but it does the region managment
> also. Am not very sure in 1.1.2 is HMaster allowed to host regions ? If so
> you need to configure MaxDirectMemory, if not probably we can see how we
> can avoid it. We need to raise a JIRA for that.
>
> Coming to the size of MaxDirectMemory less than bucket cache - I am
> wondering was there a bug previously? Because assume you need 25G offheap
> bucket cache then atleast 25G MaxDirectMemory is a must. Ideally you may
> need some delta more than 25G.
>
> 0.98 is obsolete now so its better we go with how 1.1.2 works. But if you
> feel there is a documentation that could help I think it is better we
> provide one so that users like you are not affected.
>
> Regards
> Ram
>
>
>
>
> On Tue, Sep 12, 2017 at 10:22 PM, Arul Ramachandran <arkuppai@gmail.com>
> wrote:
>
> > Thank you, Ram.
> >
> > >> So are you trying to use bucket cache feature in offheap mode with
> > 1.1.2?
> >
> > Yes.
> >
> > >> So even in 0.98 you were using bucket cache in offheap mode?
> >
> > Yes, but it is a different hbase cluster and it run 0.98. The one I am
> > trying to setup offheap cache is hbase 1.1.2
> >
> >
> > -arul
> >
> >
> > On Tue, Sep 12, 2017 at 9:34 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > Hi
> > >
> > > So are you trying to use bucket cache feature in offheap mode with
> 1.1.2?
> > > If so then it is needed that the MaxDirectMemory is greater than the
> > > offheap bucket cache size.
> > >
> > > If you are not using in offheap mode then probably there is no need for
> > > MaxDirectMemory to be greater than bucket cache size.
> > >
> > > >>in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than
> > > hbase.bucket.cache.size
> > > So even in 0.98 you were using bucket cache in offheap mode?
> > >
> > > Regards
> > > Ram
> > >
> > > On Tue, Sep 12, 2017 at 9:40 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Looks like the config you meant should be hbase.bucketcache.size
> > > >
> > > > As the refguide says:
> > > >
> > > > A float that EITHER represents a percentage of total heap memory size
> > to
> > > > give to the cache (if < 1.0) OR, it is the total capacity in
> megabytes
> > of
> > > > BucketCache. Default: 0.0
> > > >
> > > > If you specify the size as capacity, -XX:MaxDirectMemorySize should
> be
> > > > bigger than the capacity.
> > > >
> > > > For #2, did you encounter some error ?
> > > >
> > > > Cheers
> > > >
> > > > On Tue, Sep 12, 2017 at 8:52 AM, Arul Ramachandran <
> arkuppai@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > In HBase 1.1.2, I am setting up bucket cache. I set MaxDirectMemory
> > > size
> > > > > greater than hbase.bucket.cache.size  - only then it would work.
> > > > >
> > > > > 1) Does HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize needs to
be
> > > > greater
> > > > > than hbase.bucket.cache.size?
> > > > > 2) It seems with hbase 1.1.2, HBASE_MASTER_OPTS also needs the
> > > > >  -XX:MaxDirectMemorySize setting?
> > > > >
> > > > > IIRC, in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than
> > > > > hbase.bucket.cache.size --and-- I did not have to set
> > > > >  -XX:MaxDirectMemorySize for HBASE_MASTER_OPTS.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Arul
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message