hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8669) Erasure Coding: handle missing internal block locations in DFSStripedInputStream
Date Mon, 13 Jul 2015 11:40:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624549#comment-14624549
] 

Hadoop QA commented on HDFS-8669:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m 24s | Findbugs (version ) appears to be broken on
HDFS-7285. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 3 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 37s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 53s | There were no new javadoc warning messages.
|
| {color:red}-1{color} | release audit |   0m 15s | The applied patch generated 1 release
audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 37s | There were no new checkstyle issues. |
| {color:red}-1{color} | whitespace |   0m  2s | The patch has 1  line(s) that end in whitespace.
Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with eclipse:eclipse.
|
| {color:red}-1{color} | findbugs |   3m 29s | The patch appears to introduce 5 new Findbugs
(version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 135m 52s | Tests failed in hadoop-hdfs. |
| | | 178m 43s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
|   | hadoop.hdfs.TestRead |
|   | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
|   | hadoop.hdfs.TestHdfsAdmin |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
|   | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
|   | hadoop.hdfs.TestClientReportBadBlock |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.TestAppendSnapshotTruncate |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
|   | hadoop.hdfs.TestSafeModeWithStripedFile |
|   | hadoop.hdfs.TestFileAppendRestart |
|   | hadoop.cli.TestErasureCodingCLI |
|   | hadoop.hdfs.protocol.TestBlockListAsLongs |
|   | hadoop.TestRefreshCallQueue |
|   | hadoop.hdfs.TestListFilesInDFS |
|   | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
|   | hadoop.hdfs.TestMiniDFSCluster |
|   | hadoop.security.TestPermissionSymlinks |
|   | hadoop.hdfs.TestDFSRollback |
|   | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
|   | hadoop.hdfs.TestFileConcurrentReader |
|   | hadoop.hdfs.TestFileAppend2 |
|   | hadoop.hdfs.server.datanode.TestDataNodeExit |
|   | hadoop.hdfs.TestGetFileChecksum |
|   | hadoop.security.TestRefreshUserMappings |
|   | hadoop.hdfs.TestRecoverStripedFile |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
|   | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
|   | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
|   | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
|   | hadoop.hdfs.TestBlockStoragePolicy |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
|   | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
|   | hadoop.hdfs.TestCrcCorruption |
|   | hadoop.hdfs.TestDFSUpgrade |
|   | hadoop.hdfs.server.datanode.TestIncrementalBrVariations |
|   | hadoop.hdfs.TestLeaseRecovery2 |
|   | hadoop.cli.TestXAttrCLI |
|   | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
|   | hadoop.hdfs.TestDFSShell |
|   | hadoop.hdfs.client.impl.TestLeaseRenewer |
|   | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
|   | hadoop.hdfs.protocol.TestAnnotations |
|   | hadoop.hdfs.server.datanode.TestDiskError |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.security.TestPermission |
|   | hadoop.hdfs.TestIsMethodSupported |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory |
|   | hadoop.hdfs.TestFileCreationDelete |
|   | hadoop.hdfs.TestErasureCodingZones |
|   | hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
|   | hadoop.hdfs.TestDFSStorageStateRecovery |
|   | hadoop.hdfs.util.TestStripedBlockUtil |
|   | hadoop.hdfs.TestFileStatusWithECschema |
|   | hadoop.hdfs.TestAppendDifferentChecksum |
|   | hadoop.hdfs.TestRemoteBlockReader |
|   | hadoop.hdfs.TestRestartDFS |
|   | hadoop.cli.TestHDFSCLI |
|   | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage |
|   | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles |
|   | hadoop.hdfs.server.datanode.TestBlockScanner |
|   | hadoop.hdfs.TestHDFSFileSystemContract |
|   | hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation |
|   | hadoop.cli.TestAclCLI |
|   | hadoop.hdfs.TestDFSPermission |
|   | hadoop.hdfs.TestHDFSServerPorts |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
|   | hadoop.hdfs.security.TestClientProtocolWithDelegationToken |
|   | hadoop.cli.TestCryptoAdminCLI |
|   | hadoop.hdfs.server.datanode.TestStorageReport |
|   | hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | hadoop.hdfs.server.datanode.TestDeleteBlockPool |
|   | hadoop.hdfs.TestBlockReaderLocalLegacy |
|   | hadoop.hdfs.security.token.block.TestBlockToken |
|   | hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2 |
|   | hadoop.hdfs.server.datanode.TestBlockRecovery |
|   | hadoop.hdfs.TestDFSStartupVersions |
|   | hadoop.hdfs.TestWriteBlockGetsBlockLengthHint |
|   | hadoop.hdfs.TestAbandonBlock |
|   | hadoop.hdfs.TestFetchImage |
|   | hadoop.cli.TestCacheAdminCLI |
|   | hadoop.hdfs.TestDatanodeDeath |
|   | hadoop.hdfs.server.datanode.TestFsDatasetCache |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation |
|   | hadoop.hdfs.security.TestDelegationToken |
|   | hadoop.hdfs.server.datanode.TestCachingStrategy |
|   | hadoop.hdfs.TestDFSClientExcludedNodes |
|   | hadoop.hdfs.TestSnapshotCommands |
|   | hadoop.hdfs.server.datanode.TestHSync |
|   | hadoop.hdfs.TestDFSInputStream |
|   | hadoop.hdfs.TestDFSOutputStream |
|   | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.TestDFSInotifyEventInputStream |
|   | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter |
|   | hadoop.hdfs.TestReservedRawPaths |
|   | hadoop.hdfs.TestFileCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12744999/HDFS-8669-HDFS-7285.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / f4098df |
| Release Audit | https://builds.apache.org/job/PreCommit-HDFS-Build/11680/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/11680/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings | https://builds.apache.org/job/PreCommit-HDFS-Build/11680/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11680/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11680/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11680/console |


This message was automatically generated.

> Erasure Coding: handle missing internal block locations in DFSStripedInputStream
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-8669
>                 URL: https://issues.apache.org/jira/browse/HDFS-8669
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-8669-HDFS-7285.001.patch, HDFS-8669.000.patch, HDFS-8669.001.patch
>
>
> Currently DFSStripedInputStream assumes we always have complete internal block location
information, i.e., we can always get all the DataNodes for a striped block group. In a lot
of scenarios the client cannot get complete block location info, e.g., some internal blocks
are missing and the NameNode has not finished the recovery yet. We should add functionality
to handle missing block locations in DFSStripedInputStream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message