hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Mahfoud (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-2925) LRU of HConnectionManager.HBASE_INSTANCES breaks if HBaseConfiguration is changed
Date Tue, 17 Aug 2010 21:46:22 GMT

     [ https://issues.apache.org/jira/browse/HBASE-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Mahfoud updated HBASE-2925:
----------------------------------

        Fix Version/s:     (was: 0.90.0)
                           (was: 0.20.3)
                 Tags: noob
    Affects Version/s: 0.20.3
                       0.90.0
                           (was: 0.20.0)
          Description: 
{{HConnectionManager.getConnection(config)}} caches the created {{TableServer}} in {{HBASE_INSTANCES}}
(a {{LinkedHashMap}} ) which is keyed by the configuration instance itself.
Given the current implementation of {{hashCode()}} (and {{equals()}}) of {{HBaseConfiguration}},
the hash code of the configuration is changed if any of its properties are changed, which
will cause the keys of {{HBASE_INSTANCES}} to be inconsistent with the hashtable that contains
them, making some entries unreachable.
In this case, when the map's LRU strategy needs to remove the oldest entry, it tries to remove
it based on the oldest key, which no longer gives the original hash code, therefore the lookup
in {{HBASE_INSTANCES.remove(oldest)}} doesn't actually remove anything.

This has been observed to lead to OOM errors in long running clients.


  was:
HConnectionManager.HBASE_INSTANCES is a WeakHashMap from HBaseConfiguration to TableServers.
 However, each TableServers has a strong reference back to the HBaseConfiguration key so they
are never freed.  (See note at http://java.sun.com/javase/6/docs/api/java/util/WeakHashMap.html
: "Implementation note: The value objects in a WeakHashMap are held by ordinary strong references.
Thus care should be taken to ensure that value objects do not strongly refer to their own
keys, either directly or indirectly, since that will prevent the keys from being discarded.")

Moreover, HBaseConfiguration implements hashCode() but not equals() so identical HBaseConfiguration
objects each get their own TableServers object.

We had a long running HBase client process that was creating new HTable() objects, each creating
a new HBaseConfiguration() and thus a new TableServers object.  It eventually went OOM, and
gave a heap dump indicating 360 MB of data retained by HBASE_INSTANCES.


> LRU of HConnectionManager.HBASE_INSTANCES breaks if HBaseConfiguration is changed
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-2925
>                 URL: https://issues.apache.org/jira/browse/HBASE-2925
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.90.0
>            Reporter: Robert Mahfoud
>            Assignee: Dave Latham
>
> {{HConnectionManager.getConnection(config)}} caches the created {{TableServer}} in {{HBASE_INSTANCES}}
(a {{LinkedHashMap}} ) which is keyed by the configuration instance itself.
> Given the current implementation of {{hashCode()}} (and {{equals()}}) of {{HBaseConfiguration}},
the hash code of the configuration is changed if any of its properties are changed, which
will cause the keys of {{HBASE_INSTANCES}} to be inconsistent with the hashtable that contains
them, making some entries unreachable.
> In this case, when the map's LRU strategy needs to remove the oldest entry, it tries
to remove it based on the oldest key, which no longer gives the original hash code, therefore
the lookup in {{HBASE_INSTANCES.remove(oldest)}} doesn't actually remove anything.
> This has been observed to lead to OOM errors in long running clients.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message