hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
Date Wed, 28 Sep 2016 08:12:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528830#comment-15528830
] 

Hadoop QA commented on HBASE-15560:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 5 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 54s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 22s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 35s {color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue}
Skipped patched modules with no Java source: hbase-resource-bundle . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 34s {color} | {color:red}
hbase-common in master has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s {color} |
{color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 59s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 18s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 18s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 35s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s {color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 43s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2
2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 37s {color}
| {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue}
Skipped patched modules with no Java source: hbase-resource-bundle . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 39s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s {color} | {color:green}
hbase-resource-bundle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s {color} | {color:green}
hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 20s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 57s {color} | {color:red}
root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 51s {color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 233m 18s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
|   | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
|   | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
|   | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
|   | org.apache.hadoop.hbase.client.TestAdmin2 |
|   | org.apache.hadoop.hbase.client.TestHCM |
|   | org.apache.hadoop.hbase.client.TestSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830624/HBASE-15560.patch
|
| JIRA Issue | HBASE-15560 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  xml  |
| uname | Linux 4eb6ba4f664d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / de7316b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/3748/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3748/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3748/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-HBASE-Build/3748/artifact/patchprocess/patch-unit-hbase-server.txt
https://builds.apache.org/job/PreCommit-HBASE-Build/3748/artifact/patchprocess/patch-unit-root.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/3748/testReport/ |
| modules | C: hbase-resource-bundle hbase-common hbase-server . U: . |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/3748/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> TinyLFU-based BlockCache
> ------------------------
>
>                 Key: HBASE-15560
>                 URL: https://issues.apache.org/jira/browse/HBASE-15560
>             Project: HBase
>          Issue Type: Improvement
>          Components: BlockCache
>    Affects Versions: 2.0.0
>            Reporter: Ben Manes
>            Assignee: Ben Manes
>         Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
HBASE-15560.patch, tinylfu.patch
>
>
> LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and recency of
the working set. It achieves concurrency by using an O( n ) background thread to prioritize
the entries and evict. Accessing an entry is O(1) by a hash table lookup, recording its logical
access time, and setting a frequency flag. A write is performed in O(1) time by updating the
hash table and triggering an async eviction thread. This provides ideal concurrency and minimizes
the latencies by penalizing the thread instead of the caller. However the policy does not
age the frequencies and may not be resilient to various workload patterns.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the frequency
in a counting sketch, ages periodically by halving the counters, and orders entries by SLRU.
An entry is discarded by comparing the frequency of the new arrival (candidate) to the SLRU's
victim, and keeping the one with the highest frequency. This allows the operations to be performed
in O(1) time and, though the use of a compact sketch, a much larger history is retained beyond
the current working set. In a variety of real world traces the policy had [near optimal hit
rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> Concurrency is achieved by buffering and replaying the operations, similar to a write-ahead
log. A read is recorded into a striped ring buffer and writes to a queue. The operations are
applied in batches under a try-lock by an asynchronous thread, thereby track the usage pattern
without incurring high latencies ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
> In YCSB benchmarks the results were inconclusive. For a large cache (99% hit rates) the
two caches have near identical throughput and latencies with LruBlockCache narrowly winning.
At medium and small caches, TinyLFU had a 1-4% hit rate improvement and therefore lower latencies.
The lack luster result is because a synthetic Zipfian distribution is used, which SLRU performs
optimally. In a more varied, real-world workload we'd expect to see improvements by being
able to make smarter predictions.
> The provided patch implements BlockCache using the [Caffeine|https://github.com/ben-manes/caffeine]
caching library (see HighScalability [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]).
> Edward Bortnikov and Eshcar Hillel have graciously provided guidance for evaluating this
patch ([github branch|https://github.com/ben-manes/hbase/tree/tinylfu]).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message