hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
Date Thu, 22 Oct 2015 14:15:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969211#comment-14969211

Yu Li commented on HBASE-14463:

Checked multi get with --multiGet=100, and checked both 25 threads and 50 threads, results
are as follows:

w/o patch:
1. Min: 94303ms    Max: 94853ms    Avg: 94651ms
2. Min: 92547ms    Max: 93140ms    Avg: 92934ms
3. Min: 90562ms    Max: 91374ms    Avg: 91168ms
Average: 92917.67ms

w/ patch:
1. Min: 94026ms    Max: 94631ms    Avg: 94417ms
2. Min: 94086ms    Max: 94850ms    Avg: 94643ms
3. Min: 95041ms    Max: 95585ms    Avg: 95371ms
Average: 94810ms

with slow purge(lock pool size reach 5000):
1. Min: 94699ms    Max: 95276ms    Avg: 95031ms
2. Min: 97087ms    Max: 98106ms    Avg: 97871ms
3. Min: 93804ms    Max: 94743ms    Avg: 94519ms
Average: 95807ms
>From which we could see there's indeed a ~2% downgrade w/ patch, and slow purge (only
purge lockPool when its size reaches 5000) didn't bring any improvement

However, with 50 threads:
w/o patch
1. Min: 195573ms   Max: 199973ms   Avg: 199016ms
2. Min: 191126ms   Max: 194468ms   Avg: 193592ms
3. Min: 195415ms   Max: 198985ms   Avg: 198133ms
Average: 196913.67

w/ patch:
1. Min: 199279ms   Max: 202442ms   Avg: 201607ms
2. Min: 191189ms   Max: 194442ms   Avg: 193626ms
3. Min: 190235ms   Max: 193817ms   Avg: 192996ms
Average: 196076.33ms
I got a better result w/ patch.

Now my suspicion is the different distribution of chosen keys makes different result, so I'm
planning to test the 2 cases (w/ and w/o patch) with exactly the same randomly-generated keys
and see what will happen

> Severe performance downgrade when parallel reading a single key from BucketCache
> --------------------------------------------------------------------------------
>                 Key: HBASE-14463
>                 URL: https://issues.apache.org/jira/browse/HBASE-14463
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.14, 1.1.2
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>         Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch, HBASE-14463_v11.patch,
HBASE-14463_v12.patch, HBASE-14463_v2.patch, HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
TestBucketCache-new_with_IdLock.png, TestBucketCache-new_with_IdReadWriteLock.png, TestBucketCache_with_IdLock-latest.png,
TestBucketCache_with_IdLock.png, TestBucketCache_with_IdReadWriteLock-latest.png, TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
TestBucketCache_with_IdReadWriteLock.png, test-results.tar.gz
> We store feature data of online items in HBase, do machine learning on these features,
and supply the outputs to our online search engine. In such scenario we will launch hundreds
of yarn workers and each worker will read all features of one item(i.e. single rowkey in HBase),
so there'll be heavy parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc issue, and
just as titled we have observed severe performance downgrade. After some analytics we found
the root cause is the lock in BucketCache#getBlock, as shown below
> {code}
>       try {
>         lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
>         // ...
>         if (bucketEntry.equals(backingMap.get(key))) {
>           // ...
>           int len = bucketEntry.getLength();
>           Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
>               bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the operation
in LruCache. And since we're using synchronized in IdLock#getLockEntry, parallel read dropping
on the same bucket would be executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in BucketCache, and
introduce a new class called IdReadWriteLock to implement it.

This message was sent by Atlassian JIRA

View raw message