hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17747) Support both weak and soft object pool
Date Tue, 14 Mar 2017 07:43:41 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923729#comment-15923729

Yu Li commented on HBASE-17747:

bq. This is the GC options we used for CMS
Well, yes, we also have more complicated options online, but for testing, I don't think all
the configurations are relative... Still, please give it a try if you think we could resolve
it through option tuning, and feel free to revert the commit if you succeed... Or if you'd
like me to try the combination you posted and will admit that we cannot resolve the issue
by tuning if this combination couldn't work, just let me know.

bq. And still, you need to test it with G1 as the problem you described is highly related
to the GC algorithm
Notice that this is weak/soft reference object, not plain object, so we are talking about
how JVM will clean up the object with weak/soft reference automatically and the difference.
I don't think this is GC algorithm related... No matter which GC algorithm, it should clear
the object immediately when there's no other but a weak reference to it, so with weak reference
more {{IdReadWriteLock}} objects will be created, and under high read concurrency, such objects
are so many that affects GC speed (for whatever GC algorithm), while soft reference is totally
another case. See below codes in {{ObjectPool#get}}
  public V get(K key) {
    Reference<V> ref = referenceCache.get(key);
    if (ref != null) {
      V obj = ref.get();
      if (obj != null) {
        return obj;// with soft reference, we will probably return here
      referenceCache.remove(key, ref);

    V newObj = objectFactory.createObject(key); // with weak reference, we will probably create
new object here
    Reference<V> newRef = createReference(key, newObj);
    while (true) {
      Reference<V> existingRef = referenceCache.putIfAbsent(key, newRef);
      if (existingRef == null) {
        return newObj;

      V existingObject = existingRef.get();
      if (existingObject != null) {
        return existingObject;
      referenceCache.remove(key, existingRef);

> Support both weak and soft object pool
> --------------------------------------
>                 Key: HBASE-17747
>                 URL: https://issues.apache.org/jira/browse/HBASE-17747
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.0
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0
>         Attachments: HBASE-17747.patch, HBASE-17747.v2.patch, HBASE-17747.v3.patch
> During YCSB testing on embedded mode after HBASE-17744, we found that under high read
load GC is quite severe even with offheap L2 cache. After some investigation, we found it's
caused by using weak reference in {{IdReadWriteLock}}. In embedded mode the read is so quick
that the lock might already get promoted to the old generation when the weak reference is
cleared, which causes dirty card table (old reference get removed and new lock object set
into {{referenceCache}}, see {{WeakObjectPool#get}}) thus slowing YGC. In distributed mode
there'll also be more lock object created with weak reference than soft reference that slowing
down the processing.
> So we proposed to use soft reference for this {{IdReadWriteLock}} used in cache, which
won't get cleared until JVM memory is not enough, and could resolve the issue mentioned above.
What's more, we propose to extend the {{WeakObjectPool}} to be more generate to support both
weak and soft reference.
> Note that the GC issue only emerges under embedded mode with DirectOperator, in which
case all costs on the wire is removed thus produces extremely high concurrency.

This message was sent by Atlassian JIRA

View raw message