hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4150) Potentially too many connections may be opened if ThreadLocalPool or RoundRobinPool is used
Date Thu, 04 Aug 2011 05:57:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079242#comment-13079242
] 

Lars Hofhansl commented on HBASE-4150:
--------------------------------------

Having an upper safety limit on the number connections (and the accompanying threads) created
is not necessarily bad; that is especially true when the number of threads is somewhat hard
to predict, for example when using HTablePool and HTables.

The size for the threadlocal pool should just be "reasonably large" and for the round-robin
pool it should be "reasonably small".

Also, isn't PoolMap also used in HTablePool now ([HBASE-2938|https://issues.apache.org/jira/browse/HBASE-2938])?
Do we want the exact same behavior there?

> Potentially too many connections may be opened if ThreadLocalPool or RoundRobinPool is
used
> -------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4150
>                 URL: https://issues.apache.org/jira/browse/HBASE-4150
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>            Assignee: Doug Meil
>             Fix For: 0.92.0
>
>         Attachments: 4150-1.txt, 4150.txt
>
>
> See 'Problem with hbase.client.ipc.pool.type=threadlocal in trunk' discussion started
by Lars George.
> From Lars Hofhansl:
> Looking at HBaseClient.getConnection(...) I see this:
> {code}
>      synchronized (connections) {
>        connection = connections.get(remoteId);
>        if (connection == null) {
>          connection = new Connection(remoteId);
>          connections.put(remoteId, connection);
>        }
>      }
> {code}
> At the same time PoolMap.ThreadLocalPool.put is defined like this:
> {code}
>    public R put(R resource) {
>      R previousResource = get();
>      if (previousResource == null) {
> ...
>        if (poolSize.intValue() >= maxSize) {
>          return null;
>        }
> ...
>    }
> {code}
> So... If the ThreadLocalPool reaches its capacity it always returns null and hence all
new threads will create a
> new connection every time getConnection is called!
> I have also verified with a test program that works fine as long as the number of client
threads (which include
> the threads in HTable's threadpool of course) is < poolsize. Once that is no longer
the case the number of
> connections "explodes" and the program dies with OOMEs (mostly because each Connection
is associated with
> yet another thread).
> It's not clear what should happen, though. Maybe (1) the ThreadLocalPool should not have
a limit, or maybe
> (2) allocations past the pool size should throw an exception (i.e. there's a hard limit),
or maybe (3) in that case
> a single connection is returned for all threads while the pool it over its limit or (4)
we start round robin with the other
> connection in the other thread locals.
> For #1 means that the number of client threads needs to be more carefully managed by
the client app.
> In this case it would also be somewhat pointless that Connection have their own threads,
we just pass stuff
> between threads.
> #2 would work, but puts more logic in the client.
> #3 would lead to hard to debug performance issues.
> And #4 is messy :)
> From Ted Yu:
> For HBaseClient, at least the javadoc doesn't match:
> {code}
>    * @param config configuration
>    * @return either a {@link PoolType#Reusable} or {@link PoolType#ThreadLocal}
>    */
>   private static PoolType getPoolType(Configuration config) {
>     return PoolType.valueOf(config.get(HConstants.HBASE_CLIENT_IPC_POOL_TYPE),
>         PoolType.RoundRobin, PoolType.ThreadLocal);
> {code}
> I think for RoundRobinPool, we shouldn't allow maxSize to be Integer#MAX_VALUE. Otherwise
connection explosion described by Lars may incur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message