hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Helmling (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4150) Potentially too many connections may be opened if ThreadLocalPool or RoundRobinPool is used
Date Thu, 04 Aug 2011 21:45:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079619#comment-13079619
] 

Gary Helmling commented on HBASE-4150:
--------------------------------------

Thanks for the updated patch Karthick.

One additional question on how this interacts with HTablePool.  In HTablePool, when obtaining
a table:
{code}
  private HTableInterface findOrCreateTable(String tableName) {
    HTableInterface table = tables.get(tableName);
    if (table == null) {
      table = createHTable(tableName);
    }
    return table;
  }
{code}

In the case of {{ThreadLocalPool}}, it seems like there's an exposure here between when the
table is created initially and when {{ThreadLocalPool.put()}} is called to set the thread
local variable (on {{PooledHTable.close()}}).  This is independent of the issue here, so I
can open a separate JIRA, just wanted to get your thoughts on if this is a potential leak.

I'll run the updated patch through tests and commit if all passes (relative to any current
trunk breakage).

> Potentially too many connections may be opened if ThreadLocalPool or RoundRobinPool is
used
> -------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4150
>                 URL: https://issues.apache.org/jira/browse/HBASE-4150
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>            Assignee: Gary Helmling
>             Fix For: 0.92.0
>
>         Attachments: 4150-1.txt, 4150.txt, 5140-2.txt
>
>
> See 'Problem with hbase.client.ipc.pool.type=threadlocal in trunk' discussion started
by Lars George.
> From Lars Hofhansl:
> Looking at HBaseClient.getConnection(...) I see this:
> {code}
>      synchronized (connections) {
>        connection = connections.get(remoteId);
>        if (connection == null) {
>          connection = new Connection(remoteId);
>          connections.put(remoteId, connection);
>        }
>      }
> {code}
> At the same time PoolMap.ThreadLocalPool.put is defined like this:
> {code}
>    public R put(R resource) {
>      R previousResource = get();
>      if (previousResource == null) {
> ...
>        if (poolSize.intValue() >= maxSize) {
>          return null;
>        }
> ...
>    }
> {code}
> So... If the ThreadLocalPool reaches its capacity it always returns null and hence all
new threads will create a
> new connection every time getConnection is called!
> I have also verified with a test program that works fine as long as the number of client
threads (which include
> the threads in HTable's threadpool of course) is < poolsize. Once that is no longer
the case the number of
> connections "explodes" and the program dies with OOMEs (mostly because each Connection
is associated with
> yet another thread).
> It's not clear what should happen, though. Maybe (1) the ThreadLocalPool should not have
a limit, or maybe
> (2) allocations past the pool size should throw an exception (i.e. there's a hard limit),
or maybe (3) in that case
> a single connection is returned for all threads while the pool it over its limit or (4)
we start round robin with the other
> connection in the other thread locals.
> For #1 means that the number of client threads needs to be more carefully managed by
the client app.
> In this case it would also be somewhat pointless that Connection have their own threads,
we just pass stuff
> between threads.
> #2 would work, but puts more logic in the client.
> #3 would lead to hard to debug performance issues.
> And #4 is messy :)
> From Ted Yu:
> For HBaseClient, at least the javadoc doesn't match:
> {code}
>    * @param config configuration
>    * @return either a {@link PoolType#Reusable} or {@link PoolType#ThreadLocal}
>    */
>   private static PoolType getPoolType(Configuration config) {
>     return PoolType.valueOf(config.get(HConstants.HBASE_CLIENT_IPC_POOL_TYPE),
>         PoolType.RoundRobin, PoolType.ThreadLocal);
> {code}
> I think for RoundRobinPool, we shouldn't allow maxSize to be Integer#MAX_VALUE. Otherwise
connection explosion described by Lars may incur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message