hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9500) datanodesSoftwareVersions map may counting wrong when rolling upgrade
Date Wed, 26 Oct 2016 18:44:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609303#comment-15609303
] 

Hadoop QA commented on HDFS-9500:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 20s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 40s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 58s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 35s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  8s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 14s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  8s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 49s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m  0s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 56s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 56s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 30s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  5s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 12s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  9s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 42s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 26s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 21s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 37s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
|   | org.apache.hadoop.hdfs.TestFileChecksum |
|   | org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead |
|   | org.apache.hadoop.hdfs.server.namenode.TestStartup |
|   | org.apache.hadoop.hdfs.TestWriteRead |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
|   | org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
|   | org.apache.hadoop.hdfs.TestFileCreationClient |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
|   | org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
|   | org.apache.hadoop.hdfs.TestMaintenanceState |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
|   | org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl |
|   | org.apache.hadoop.fs.TestEnhancedByteBufferAccess |
|   | org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete |
|   | org.apache.hadoop.hdfs.TestDFSClientFailover |
|   | org.apache.hadoop.hdfs.TestSetrepIncreasing |
|   | org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
|   | org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | org.apache.hadoop.hdfs.server.namenode.TestNameNodeRecovery |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
|   | org.apache.hadoop.hdfs.TestAclsEndToEnd |
|   | org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
|   | org.apache.hadoop.hdfs.TestFileCreation |
|   | org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
|   | org.apache.hadoop.hdfs.TestReplication |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
|   | org.apache.hadoop.hdfs.server.namenode.TestINodeFile |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLog |
|   | org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeLifeline |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | org.apache.hadoop.hdfs.TestSafeModeWithStripedFile |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
|   | org.apache.hadoop.hdfs.TestReconstructStripedFile |
|   | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
|   | org.apache.hadoop.hdfs.server.datanode.TestLargeBlockReport |
|   | org.apache.hadoop.hdfs.server.namenode.TestBackupNode |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSImage |
|   | org.apache.hadoop.hdfs.TestDFSClientRetries |
|   | org.apache.hadoop.hdfs.server.namenode.TestQuotaByStorageType |
|   | org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
|   | org.apache.hadoop.hdfs.TestDFSStorageStateRecovery |
|   | org.apache.hadoop.hdfs.TestFileAppend3 |
|   | org.apache.hadoop.hdfs.TestHDFSFileSystemContract |
|   | org.apache.hadoop.hdfs.TestFileAppend2 |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
|   | org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults |
|   | org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSImageWithXAttr |
|   | org.apache.hadoop.hdfs.TestDFSPermission |
|   | org.apache.hadoop.hdfs.TestDecommission |
|   | org.apache.hadoop.hdfs.server.datanode.TestBatchIbr |
|   | org.apache.hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
|   | org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives |
|   | org.apache.hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
|   | org.apache.hadoop.hdfs.security.TestDelegationToken |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStream |
|   | org.apache.hadoop.hdfs.client.impl.TestBlockReaderLocal |
|   | org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
|   | org.apache.hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | org.apache.hadoop.hdfs.server.namenode.TestFsck |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing |
|   | org.apache.hadoop.hdfs.TestCrcCorruption |
|   | org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
|   | org.apache.hadoop.hdfs.TestLeaseRecovery |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
|   | org.apache.hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
|   | org.apache.hadoop.hdfs.server.namenode.TestFileTruncate |
|   | org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
|   | org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles |
|   | org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
|   | org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode |
|   | org.apache.hadoop.hdfs.server.mover.TestStorageMover |
|   | org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks |
|   | org.apache.hadoop.hdfs.server.balancer.TestBalancer |
|   | org.apache.hadoop.hdfs.TestDatanodeReport |
|   | org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
|   | org.apache.hadoop.hdfs.TestAppendSnapshotTruncate |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean |
|   | org.apache.hadoop.hdfs.server.namenode.TestCheckpoint |
|   | org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean |
|   | org.apache.hadoop.hdfs.server.namenode.TestAddStripedBlocks |
|   | org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | org.apache.hadoop.hdfs.TestDataStream |
|   | org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
|   | org.apache.hadoop.hdfs.server.mover.TestMover |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
|   | org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
|   | org.apache.hadoop.hdfs.server.namenode.TestNestedEncryptionZones |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
|   | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
|   | org.apache.hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC |
|   | org.apache.hadoop.hdfs.TestRenameWhileOpen |
|   | org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader |
|   | org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
|   | org.apache.hadoop.hdfs.TestFileConcurrentReader |
|   | org.apache.hadoop.hdfs.server.datanode.TestBPOfferService |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReconstruction |
|   | org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
|   | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
|
|   | org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat |
|   | org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
|   | org.apache.hadoop.hdfs.server.namenode.TestDeleteRace |
|   | org.apache.hadoop.hdfs.TestDecommissionWithStriped |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogRace |
|   | org.apache.hadoop.hdfs.server.namenode.TestAddBlock |
|   | org.apache.hadoop.hdfs.server.datanode.TestBlockScanner |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
|   | org.apache.hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
|   | org.apache.hadoop.hdfs.TestDistributedFileSystem |
|   | org.apache.hadoop.fs.TestSWebHdfsFileContextMainOperations |
|   | org.apache.hadoop.hdfs.TestHFlush |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
|   | org.apache.hadoop.fs.TestSymlinkHdfsFileSystem |
|   | org.apache.hadoop.hdfs.server.namenode.TestAuditLogs |
|   | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
|   | org.apache.hadoop.hdfs.server.namenode.TestHostsFiles |
|   | org.apache.hadoop.hdfs.TestGetBlocks |
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode |
|   | org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
|   | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
|   | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9500 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12835385/HDFS-9500.002.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux f77703842fb8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f209e93 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17294/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17294/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17294/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> datanodesSoftwareVersions map may counting wrong when rolling upgrade
> ---------------------------------------------------------------------
>
>                 Key: HDFS-9500
>                 URL: https://issues.apache.org/jira/browse/HDFS-9500
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.1, 2.6.2
>            Reporter: Phil Yang
>            Assignee: Erik Krogen
>         Attachments: 9500-v1.patch, HDFS-9500.000.patch, HDFS-9500.001.patch, HDFS-9500.002.patch
>
>
> While rolling upgrading, namenode's website overview will report there are two versions
datanodes in the cluster, for example, 2.6.0 has x nodes and 2.6.2 has y nodes. However, sometimes
when I stop a datanode in old version and start a new version one, namenode only increases
the number of new version but not decreases the number of old version. So the total number
x+y will be larger than the number of datanodes. Even all datanodes are upgraded, there will
still have the messages that there are several datanode in old version. And I must run hdfs
dfsadmin -refreshNodes to clear this message.
> I think this issue is caused by DatanodeManager.registerDatanode. If nodeS in old version
is not alive because of shutting down, it will not pass shouldCountVersion, so the number
of old version won't be decreased. But this method only judges the status of heartbeat and
isAlive at that moment, if namenode has not removed this node which will decrease the version
map and this node restarts in the new version, the decrementVersionCount belongs to this node
will never be executed.
> So the simplest way to fix this is that we always recounting the version map in registerDatanode
since it is not a heavy operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message