hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben West <bwsithspaw...@yahoo.com>
Subject Re: Meta region hotspotting
Date Fri, 23 Dec 2011 21:42:14 GMT
For those following at home, I tried this change and it does not appear to have solved the
problem. Back to the drawing board.


----- Original Message -----
From: Stack <saint.ack@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
Sent: Thursday, December 15, 2011 7:08 PM
Subject: Re: Meta region hotspotting

Can u try w/o a limit or with upped limit and see if a diff?    Sounds plausible yes but
looks like u could make a fact with some small experiments

Thanks



On Dec 15, 2011, at 12:25 PM, Ben West <bwsithspawn00@yahoo.com> wrote:

> Digging into this further, I see the following in HTablePool:
> 
>   public void putTable(HTableInterface table) {
>     LinkedList<HTableInterface> queue = tables.get(Bytes.toString(table.getTableName()));
>     synchronized(queue) {
>       if(queue.size() >= maxSize) {
>         // release table instance since we're not reusing it
>         this.tableFactory.releaseHTableInterface(table);
>         return;
>       }
>       queue.add(table);
>     }
>   }
> 
> maxSize is hard coded to 10, so I think if there are more than 10 simultaneous requests,
future requests will get brand-new tables, lacking any cache. 
> 
> Does this sound plausible? I can submit a JIRA asking for maxSize to be a config param
if so. We have >> 10 simultaneous requests.
> 
> 
> ----- Original Message -----
> From: Ben West <bwsithspawn00@yahoo.com>
> To: Stack <stack@duboce.net>; "user@hbase.apache.org" <user@hbase.apache.org>
> Cc: 
> Sent: Friday, December 9, 2011 10:52 AM
> Subject: Re: Meta region hotspotting
> 
> Thanks Stack.
> 
> Is there any way I can log requests going to .META. vs. region servers? I've tried setting
various portions of HBase to DEBUG but I haven't found the magic combo yet.
> 
> 
> ----- Original Message -----
> From: Stack <stack@duboce.net>
> To: user@hbase.apache.org; Ben West <bwsithspawn00@yahoo.com>
> Cc: 
> Sent: Thursday, December 8, 2011 6:24 PM
> Subject: Re: Meta region hotspotting
> 
> On Thu, Dec 8, 2011 at 4:05 PM, Ben West <bwsithspawn00@yahoo.com> wrote:
>> We have a cluster with four region servers and about 2,000 regions. We're using the
REST server, and we've noticed that whatever region is hosting META gets 3-5x the number of
requests that the other regions do.
>> 
>> It's my understanding that the client should cache the row start/end locations, and
hence should need to go to META only rarely after a little bit. But I'm sustaining tens of
thousands of requests per second for many minutes and they all still seem to hit META - does
anyone know why this would be?
>> 
> 
> Your understanding is corrrect.  I'd think that the caching would
> eventually pick up all 2k region locations  unless REST is making a
> new client on every invocation, or every few invocations (I've not
> checked).
> 
> St.Ack


Mime
View raw message