hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "genericqa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
Date Wed, 17 Jan 2018 04:38:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328260#comment-16328260
] 

genericqa commented on HDFS-13024:
----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 49s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m  8s{color}
| {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  2s{color} |
{color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 43s{color}
| {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  7s{color} |
{color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 17s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 13s{color} |
{color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  2s{color} |
{color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m  6s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 59s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 59s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 38s{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 0 unchanged
- 2 fixed = 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  5s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 29s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m 21s{color} | {color:red}
hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  4s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 18s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 23s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 24s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Synchronization performed on java.util.concurrent.ConcurrentHashMap in org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.triggerPostContainerCreate(String,
Message)  At ContainerStateMachine.java:org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.triggerPostContainerCreate(String,
Message)  At ContainerStateMachine.java:[line 184] |
|  |  Synchronization performed on java.util.concurrent.ConcurrentHashMap in org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(RaftProtos$LogEntryProto)
 At ContainerStateMachine.java:org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(RaftProtos$LogEntryProto)
 At ContainerStateMachine.java:[line 167] |
| Failed junit tests | hadoop.cblock.TestCBlockCLI |
|   | hadoop.hdfs.tools.TestDFSZKFailoverController |
|   | hadoop.ozone.scm.TestSCMCli |
|   | hadoop.ozone.TestStorageContainerManager |
|   | hadoop.ozone.container.common.impl.TestContainerPersistence |
|   | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
|   | hadoop.ozone.client.rpc.TestOzoneRpcClient |
|   | hadoop.ozone.TestOzoneConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13024 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12906278/HDFS-13024-HDFS-7240.001.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient
 findbugs  checkstyle  |
| uname | Linux 92371384abf5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 18f9fea |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/22672/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/22672/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/22672/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/22672/testReport/ |
| Max. process+thread count | 4081 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/22672/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone: ContainerStateMachine should synchronize operations between createContainer op
and writeChunk
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-13024
>                 URL: https://issues.apache.org/jira/browse/HDFS-13024
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>            Priority: Major
>             Fix For: HDFS-7240
>
>         Attachments: HDFS-13024-HDFS-7240.001.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been divided
into two stages 1) the log write phase (here the state machine data is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log and forward
it to its followers. However there is no guarantee on when the followers will apply the log
to the state machine in {{applyTransaction}}. 
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please note that
the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however the create
container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message