hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
Date Thu, 07 Jan 2016 04:34:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086787#comment-15086787
] 

Hadoop QA commented on HADOOP-12678:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 8s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 15s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s {color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 8s {color} | {color:red}
Patch generated 2 new checkstyle issues in hadoop-tools/hadoop-azure (total was 25, now 27).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 35s {color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 59s {color} | {color:red}
hadoop-tools_hadoop-azure-jdk1.8.0_66 with JDK v1.8.0_66 generated 18 new issues (was 26,
now 26). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s {color} | {color:green}
the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 26s {color} | {color:red}
hadoop-tools_hadoop-azure-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 1, now
1). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s {color} | {color:green}
hadoop-azure in the patch passed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s {color} | {color:green}
hadoop-azure in the patch passed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 44s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12780895/HADOOP-12678.006.patch
|
| JIRA Issue | HADOOP-12678 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 25de3887313a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 791c163 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
|
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/8351/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| javadoc | hadoop-tools_hadoop-azure-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8351/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdk1.8.0_66.txt
|
| javadoc | hadoop-tools_hadoop-azure-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8351/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdk1.7.0_91.txt
|
| JDK v1.7.0_91  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8351/testReport/
|
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Max memory used | 76MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8351/console |


This message was automatically generated.



> Handle empty rename pending metadata file during atomic rename in redo path
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-12678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12678
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/azure
>            Reporter: madhumita chakraborty
>            Assignee: madhumita chakraborty
>            Priority: Critical
>         Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, HADOOP-12678.003.patch,
HADOOP-12678.004.patch, HADOOP-12678.005.patch, HADOOP-12678.006.patch
>
>
> Handle empty rename pending metadata file during atomic rename in redo path
> During atomic rename we create metadata file for rename(-renamePending.json). We create
that in 2 steps
> 1. We create an empty blob corresponding to the .json file in its real location
> 2. We create a scratch file to which we write the contents of the rename pending which
is then copied over into the blob described in 1
> If process crash occurs after step 1 and before step 2 is complete - we will be left
with a zero size blob corresponding to the pending rename metadata file.
> This kind of scenario can happen in the /hbase/.tmp folder because it is considered a
candidate folder for atomic rename. Now when HMaster starts up it executes listStatus on the
.tmp folder to clean up pending data. At this stage due to the lazy pending rename complete
process we look for these json files. On seeing an empty file the process simply throws a
fatal exception assuming something went wrong.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message