hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Helmling <ghelml...@gmail.com>
Subject Re: Allocating more heap for endpoint coprocessors
Date Fri, 31 Aug 2012 15:59:33 GMT
Maybe we need to add a coprocessors section to the ref guide.  I think
all the current documentation is in javadoc.  And if all the
potentially destabilizing issues of in-process coprocessor usage are
not yet called out (memory usage, cpu, etc), we could more explicitly
detail that.

In we want to really be able to support these cases, though, we
probably need to push HBASE-4047 (external coprocessor host) forward.
Without it there's not a lot we can do to prevent coprocessors from
taking down region servers.

On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <lhofhansl@yahoo.com> wrote:
> Maybe we should be better with marking improvement as such in jira, and then have a special
list in the release announcements highlighting these (otherwise they just be in the noise
of all the other bug fixes).
> -- Lars
> ________________________________
>  From: Ramkrishna.S.Vasudevan <ramkrishna.vasudevan@huawei.com>
> To: user@hbase.apache.org; 'lars hofhansl' <lhofhansl@yahoo.com>
> Sent: Thursday, August 30, 2012 9:59 PM
> Subject: RE: Allocating more heap for endpoint coprocessors
> @Lars/@Gary
> Do we need to document such things.  Recently someone was asking me a
> question like this, if my endpoint impl is so memory intensive it just
> affects a running cluster and already the RS
> has a huge memory heap associated with it.
> So its better we document saying if your endpoint consumes memory and
> because it runs along with RS based on your need add that extra amount of
> memory that will be taken up by the endpoint impl.
> Regards
> Ram
>> -----Original Message-----
>> From: lars hofhansl [mailto:lhofhansl@yahoo.com]
>> Sent: Friday, August 31, 2012 3:04 AM
>> To: user@hbase.apache.org
>> Subject: Re: Allocating more heap for endpoint coprocessors
>> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
>> you to share state between RegionObservers (and Endpoints) within the
>> same RegionServer.
>> -- Lars
>> ________________________________
>>  From: Gary Helmling <ghelmling@gmail.com>
>> To: user@hbase.apache.org
>> Sent: Thursday, August 30, 2012 1:59 PM
>> Subject: Re: Allocating more heap for endpoint coprocessors
>> Endpoint coprocessors are loaded and run within the HBase RegionServer
>> process.  Your endpoint coprocessors will be running on the region
>> servers hosting the regions for the table(s) on which the coprocessor
>> is configured.
>> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
>> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
>> the region server.
>> Note that a separate coprocessor instance is loaded for each table
>> region, so, say you want to allocate 10MB for your coprocessor, but
>> each region server hosts 20 regions, you would want to increase the
>> heap size by 200MB (20x10MB).
>> --gh
>> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <ykim@crunchyroll.com>
>> wrote:
>> > Hi,
>> >
>> > We have some memory intensive endpoint coprocessors running on our
>> RegionServers. As a result, we want to allocate more heap for the
>> coprocessors, but there doesn't seem to be much documentation on which
>> Hbase processes are directly responsible for coprocessors. Does anyone
>> happen to know or direct me to some resource that does?
>> >
>> > Thanks,
>> > Young Kim
>> >

View raw message