hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Mahfoud (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2925) LRU of HConnectionManager.HBASE_INSTANCES breaks if HBaseConfiguration is changed
Date Fri, 03 Sep 2010 17:52:32 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905987#action_12905987
] 

Robert Mahfoud commented on HBASE-2925:
---------------------------------------

@stack, thank you for picking up this issue. Feel free to reuse the code provided in SimpleHConnectionManagerLeakReplicator.java
in any way you want. (Let me know if this is not enough I could submit another file with Apache
license added.)

Back to the discussion, I agree that removing the {{hasCode()}} and {{equals()}} methods does
resolve this issue. However, I wonder if it defeats the purpose of having a connection cache
from the first place. You also alluded to the problem of the configuration settings changing
after the connection is open, which, if we rely solely on Object equality, could lead to subtle
error or unexpected behavior.

Hence my suggestion to pin down the configuration once the TableServers instance is created.
The disadvantage there is that small/unrelated changes in the configuration will invalidate
the cache entry as well.

If that is deemed a real problem given the common use cases, I was thinking maybe we define
"equality" of the configuration for the purposes of caching as the equality of all properties
that start with {{hbase.}} or {{zk.}} or any other prefix whose properties could be used as
possible connection parameters. Not sure if the comparison becomes too slow in that case,
but what is more important is whether correctness is compromised if we make the assumption
that only properties starting with those prefixes affect the connection.

> LRU of HConnectionManager.HBASE_INSTANCES breaks if HBaseConfiguration is changed
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-2925
>                 URL: https://issues.apache.org/jira/browse/HBASE-2925
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.90.0
>            Reporter: Robert Mahfoud
>         Attachments: 2925.txt, SimpleHConnectionManagerLeakReplicator.java
>
>
> {{HConnectionManager.getConnection(config)}} caches the created {{TableServer}} in {{HBASE_INSTANCES}}
(a {{LinkedHashMap}} ) which is keyed by the configuration instance itself.
> Given the current implementation of {{hashCode()}} (and {{equals()}}) of {{HBaseConfiguration}},
the hash code of the configuration is changed if any of its properties are changed, which
will cause the keys of {{HBASE_INSTANCES}} to be inconsistent with the hashtable that contains
them, making some entries unreachable.
> In this case, when the map's LRU strategy needs to remove the oldest entry, it tries
to remove it based on the oldest key, which no longer gives the original hash code, therefore
the lookup in {{HBASE_INSTANCES.remove(oldest)}} doesn't actually remove anything.
> This has been observed to lead to OOM errors in long running clients.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message