hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17747) Support both weak and soft object pool
Date Tue, 14 Mar 2017 06:32:41 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923660#comment-15923660
] 

Yu Li commented on HBASE-17747:
-------------------------------

bq. Again, could you please share your GC options?
Sure, nothing special actually...
{noformat}-XX:+UseConcMarkSweepGC -Xmx40g -Xms40g -Xmn8g -XX:SurvivorRatio=2{noformat}
And we tried to tune Xmn and SurvivorRation but no combination worked. Just try the embedded
mode out, I bet you'll see the issue I describe here that cannot be fixed through tuning GC
options.

bq. Weak reference is also suitable here in theory...
But only in theory, and in real world the dream breaks... And with soft reference we have
5%-7% perf bump up even in distributed mode...
Let me list the commits I tried to make it live with weak reference (but failed) besides tuning
GC options:
{noformat}
commit 98a6c742ed4f5a8307f40df2bd95504ce1eb0444
Author: Yu Li <jueding.ly@alibaba-inc.com>
Date:   Thu Dec 29 17:06:59 2016 +0800

    Try to accelerate weak reference purge for WeakObjectPool

    Introduced a blocking queue, one producer to read from ReferenceQueue and enqueue, and
multiple consumers
    read from blocking queue then remove from ConcurrentHashMap in parallel

commit 0e237d345b9c6b9fbc9edd597b727139b172a15a
Author: Yu Li <jueding.ly@alibaba-inc.com>
Date:   Thu Dec 29 15:08:28 2016 +0800

    Add support of using exclusive thread for reference purge in WeakObjectPool and use it
in IdReadWriteLock (addendum)

commit 22add08c423f07058a2e231cec87806878c0d513
Author: Yu Li <jueding.ly@alibaba-inc.com>
Date:   Thu Dec 29 14:49:24 2016 +0800

    Add support of using exclusive thread for reference purge in WeakObjectPool and use it
in IdReadWriteLock

commit a9e1c9bef1b94e56d6ddda5fcbe7d7c52f5e9da4
Author: Yu Li <jueding.ly@alibaba-inc.com>
Date:   Thu Dec 29 11:22:39 2016 +0800

    Reduce invocation rate of WeakObjectPool#purge in IdReadWriteLock#getLock to avoid contention
{noformat}
So I did tried before using soft reference...

> Support both weak and soft object pool
> --------------------------------------
>
>                 Key: HBASE-17747
>                 URL: https://issues.apache.org/jira/browse/HBASE-17747
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.0
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0
>
>         Attachments: HBASE-17747.patch, HBASE-17747.v2.patch, HBASE-17747.v3.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under high read
load GC is quite severe even with offheap L2 cache. After some investigation, we found it's
caused by using weak reference in {{IdReadWriteLock}}. In embedded mode the read is so quick
that the lock might already get promoted to the old generation when the weak reference is
cleared, which causes dirty card table (old reference get removed and new lock object set
into {{referenceCache}}, see {{WeakObjectPool#get}}) thus slowing YGC. In distributed mode
there'll also be more lock object created with weak reference than soft reference that slowing
down the processing.
> So we proposed to use soft reference for this {{IdReadWriteLock}} used in cache, which
won't get cleared until JVM memory is not enough, and could resolve the issue mentioned above.
What's more, we propose to extend the {{WeakObjectPool}} to be more generate to support both
weak and soft reference.
> Note that the GC issue only emerges under embedded mode with DirectOperator, in which
case all costs on the wire is removed thus produces extremely high concurrency.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message