hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19358) Improve the stability of splitting log when do fail over
Date Fri, 29 Dec 2017 11:32:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306209#comment-16306209
] 

Hadoop QA commented on HBASE-19358:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 12s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  0s{color}
| {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 2 new or modified test files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 13s{color}
| {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 38s{color} |
{color:green} branch-1 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 39s{color} |
{color:green} branch-1 passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 18s{color}
| {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 55s{color}
| {color:green} branch has no errors when building our shaded downstream artifacts. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 12s{color} |
{color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 33s{color} |
{color:green} branch-1 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 35s{color} |
{color:green} branch-1 passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 57s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 39s{color} |
{color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 39s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 39s{color} |
{color:green} the patch passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 39s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 18s{color} | {color:red}
hbase-server: The patch generated 5 new + 67 unchanged - 2 fixed = 72 total (was 69) {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 37s{color}
| {color:green} patch has no errors when building our shaded downstream artifacts. {color}
|
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 10m 19s{color}
| {color:green} Patch does not cause any errors with Hadoop 2.4.1 2.5.2 2.6.5 2.7.4. {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m 34s{color} | {color:red}
hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 27s{color} |
{color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 36s{color} |
{color:green} the patch passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m  8s{color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 18s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 20s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
|  |  Invocation of toString on WALSplitter$RegionEntryBuffer.encodedRegionName in org.apache.hadoop.hbase.wal.WALSplitter$BoundedLogWriterCreationOutputSink.executeCloseTask(CompletionService,
List, List)  At WALSplitter.java:in org.apache.hadoop.hbase.wal.WALSplitter$BoundedLogWriterCreationOutputSink.executeCloseTask(CompletionService,
List, List)  At WALSplitter.java:[line 1745] |
| Failed junit tests | hadoop.hbase.client.TestAdmin1 |
|   | hadoop.hbase.util.TestHBaseFsck |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19358 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12904002/HBASE-19358-branch-1.patch
|
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  hadoopcheck  hbaseanti
 checkstyle  compile  |
| uname | Linux 42850578e9fa 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-1 / 528eb10 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_161 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_152 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_161
|
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/10786/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/10786/artifact/patchprocess/new-findbugs-hbase-server.html
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/10786/artifact/patchprocess/patch-unit-hbase-server.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/10786/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/10786/console |
| Powered by | Apache Yetus 0.6.0   http://yetus.apache.org |


This message was automatically generated.



> Improve the stability of splitting log when do fail over
> --------------------------------------------------------
>
>                 Key: HBASE-19358
>                 URL: https://issues.apache.org/jira/browse/HBASE-19358
>             Project: HBase
>          Issue Type: Improvement
>          Components: MTTR
>    Affects Versions: 0.98.24
>            Reporter: Jingyun Tian
>            Assignee: Jingyun Tian
>         Attachments: HBASE-18619-branch-2.patch, HBASE-19358-branch-1.patch, HBASE-19358-v1.patch,
HBASE-19358-v4.patch, HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, HBASE-19358-v8.patch,
HBASE-19358.patch
>
>
> The way we splitting log now is like the following figure:
> !https://issues.apache.org/jira/secure/attachment/12902997/split-logic-old.jpg!
> The problem is the OutputSink will write the recovered edits during splitting log, which
means it will create one WriterAndPath for each region and retain it until the end. If the
cluster is small and the number of regions per rs is large, it will create too many HDFS streams
at the same time. Then it is prone to failure since each datanode need to handle too many
streams.
> Thus I come up with a new way to split log.  
> !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg!
> We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, we will
pick the largest EntryBuffer and write it to a file (close the writer after finish). Then
after we read all entries into memory, we will start a writeAndCloseThreadPool, it starts
a certain number of threads to write all buffers to files. Thus it will not create HDFS streams
more than *_hbase.regionserver.hlog.splitlog.writer.threads_* we set.
> The biggest benefit is we can control the number of streams we create during splitting
log, 
> it will not exceeds *_hbase.regionserver.wal.max.splitters * hbase.regionserver.hlog.splitlog.writer.threads_*,
but before it is *_hbase.regionserver.wal.max.splitters * the number of region the hlog contains_*.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message