hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jingcheng Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
Date Tue, 22 Sep 2015 21:38:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903489#comment-14903489
] 

Jingcheng Du commented on HBASE-14463:
--------------------------------------

Thanks [~carp84]!
We have two places to use IdLock in mob, MobFileCache and HMobStore, where IdLock is used
as a write lock. If the performance of IdReadWriteLock can be improved in write mode, I think
you can use IdReadWriteLock in mob as well.
In MobFileCache, the evict is not to evict blocks from the cache, we just evict the un-referenced
file reader from the cache. It's ok to evict when reading.
Besides, you remove the loop in getLockEntry, and remove sync from both getLockEntry and releaseLockEntry,
what if a race condition in these methods of IdReadWriteLock, a thread acquires a write lock
but it is removed from the map by another thread because of a race condition(the code {code}entry.readWriteLock.hasQueuedThreads(){code}
and {code}boolean removeSucceed = map.remove(entry.id, entry){code} in releaseLockEntry give
the race condition a chance). It is possible, right?

> Severe performance downgrade when parallel reading a single key from BucketCache
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-14463
>                 URL: https://issues.apache.org/jira/browse/HBASE-14463
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.14, 1.1.2
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0, 1.3.0
>
>         Attachments: HBASE-14463.patch, TestBucketCache_with_IdLock.png, TestBucketCache_with_IdReadWriteLock.png
>
>
> We store feature data of online items in HBase, do machine learning on these features,
and supply the outputs to our online search engine. In such scenario we will launch hundreds
of yarn workers and each worker will read all features of one item(i.e. single rowkey in HBase),
so there'll be heavy parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc issue, and
just as titled we have observed severe performance downgrade. After some analytics we found
the root cause is the lock in BucketCache#getBlock, as shown below
> {code}
>       try {
>         lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
>         // ...
>         if (bucketEntry.equals(backingMap.get(key))) {
>           // ...
>           int len = bucketEntry.getLength();
>           Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
>               bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the operation
in LruCache. And since we're using synchronized in IdLock#getLockEntry, parallel read dropping
on the same bucket would be executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in BucketCache, and
introduce a new class called IdReadWriteLock to implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message