hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4486) Add log category for long-running DFSClient notices
Date Fri, 22 Aug 2014 02:45:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106383#comment-14106383
] 

Hadoop QA commented on HDFS-4486:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12663539/hdfs-4486-20140821-2.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.hdfs.TestReservedRawPaths
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
                  org.apache.hadoop.hdfs.TestSetrepIncreasing
                  org.apache.hadoop.hdfs.TestModTime
                  org.apache.hadoop.hdfs.security.TestDelegationToken
                  org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
                  org.apache.hadoop.hdfs.TestDisableConnCache
                  org.apache.hadoop.hdfs.server.namenode.TestEditLogAutoroll
                  org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
                  org.apache.hadoop.hdfs.TestConnCache
                  org.apache.hadoop.hdfs.TestDFSClientRetries
                  org.apache.hadoop.hdfs.TestSetrepDecreasing
                  org.apache.hadoop.hdfs.server.datanode.TestDiskError
                  org.apache.hadoop.hdfs.server.namenode.TestHostsFiles
                  org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
                  org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
                  org.apache.hadoop.hdfs.server.namenode.TestDeleteRace
                  org.apache.hadoop.hdfs.TestPread
                  org.apache.hadoop.hdfs.server.namenode.TestLeaseManager
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
                  org.apache.hadoop.hdfs.server.datanode.TestStorageReport
                  org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
                  org.apache.hadoop.hdfs.TestReadWhileWriting
                  org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
                  org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
                  org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
                  org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
                  org.apache.hadoop.hdfs.server.namenode.TestAuditLogger
                  org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
                  org.apache.hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
                  org.apache.hadoop.hdfs.TestBlocksScheduledCounter
                  org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
                  org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
                  org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy
                  org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeInitStorage
                  org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
                  org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
                  org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
                  org.apache.hadoop.hdfs.security.token.block.TestBlockToken
                  org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
                  org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount
                  org.apache.hadoop.hdfs.TestFileAppend
                  org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
                  org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
                  org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
                  org.apache.hadoop.hdfs.server.datanode.TestHSync
                  org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
                  org.apache.hadoop.hdfs.TestBlockMissingException
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeRpcServer
                  org.apache.hadoop.hdfs.TestDatanodeRegistration
                  org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
                  org.apache.hadoop.hdfs.TestAbandonBlock
                  org.apache.hadoop.hdfs.server.namenode.TestSequentialBlockId
                  org.apache.hadoop.hdfs.TestFileCreationClient
                  org.apache.hadoop.hdfs.TestDFSAddressConfig
                  org.apache.hadoop.hdfs.TestFileAppendRestart
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
                  org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl
                  org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
                  org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
                  org.apache.hadoop.hdfs.TestDFSShellGenericOptions
                  org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
                  org.apache.hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir
                  org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
                  org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy
                  org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
                  org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
                  org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
                  org.apache.hadoop.hdfs.TestReplication
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
                  org.apache.hadoop.hdfs.TestFileConcurrentReader
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
                  org.apache.hadoop.hdfs.qjournal.TestNNWithQJM
                  org.apache.hadoop.hdfs.TestClose
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit
                  org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
                  org.apache.hadoop.hdfs.TestDFSRollback
                  org.apache.hadoop.hdfs.TestFileCreationDelete
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
                  org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
                  org.apache.hadoop.hdfs.TestFileAppend4
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
                  org.apache.hadoop.hdfs.TestDFSOutputStream
                  org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
                  org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean
                  org.apache.hadoop.hdfs.TestBalancerBandwidth
                  org.apache.hadoop.hdfs.TestHdfsAdmin
                  org.apache.hadoop.hdfs.TestDFSPermission
                  org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
                  org.apache.hadoop.hdfs.TestSafeMode
                  org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
                  org.apache.hadoop.hdfs.TestDatanodeBlockScanner
                  org.apache.hadoop.hdfs.TestDFSRename
                  org.apache.hadoop.hdfs.TestIsMethodSupported
                  org.apache.hadoop.hdfs.TestSeekBug
                  org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
                  org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
                  org.apache.hadoop.hdfs.TestParallelRead
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
                  org.apache.hadoop.hdfs.TestDFSUpgrade
                  org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
                  org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
                  org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
                  org.apache.hadoop.hdfs.TestDecommission
                  org.apache.hadoop.hdfs.TestCrcCorruption
                  org.apache.hadoop.hdfs.server.namenode.TestINodeFile
                  org.apache.hadoop.hdfs.TestFSOutputSummer
                  org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
                  org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart
                  org.apache.hadoop.hdfs.TestSetTimes
                  org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
                  org.apache.hadoop.hdfs.TestHDFSFileSystemContract
                  org.apache.hadoop.hdfs.TestDFSRemove
                  org.apache.hadoop.hdfs.TestEncryptionZones
                  org.apache.hadoop.hdfs.TestAppendDifferentChecksum
                  org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
                  org.apache.hadoop.hdfs.server.namenode.ha.TestNNHealthCheck
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
                  org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
                  org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
                  org.apache.hadoop.hdfs.TestDFSFinalize
                  org.apache.hadoop.hdfs.TestFileAppend2
                  org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
                  org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports
                  org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
                  org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
                  org.apache.hadoop.hdfs.server.namenode.TestEditLog
                  org.apache.hadoop.hdfs.server.namenode.ha.TestHAFsck
                  org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag
                  org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
                  org.apache.hadoop.hdfs.TestFsShellPermission
                  org.apache.hadoop.hdfs.TestRestartDFS
                  org.apache.hadoop.hdfs.TestFileAppend3
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
                  org.apache.hadoop.hdfs.TestQuota
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean
                  org.apache.hadoop.hdfs.TestClientBlockVerification
                  org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
                  org.apache.hadoop.hdfs.TestDFSClientFailover
                  org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
                  org.apache.hadoop.hdfs.TestRollingUpgrade
                  org.apache.hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure
                  org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
                  org.apache.hadoop.hdfs.TestGetBlocks
                  org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
                  org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController
                  org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
                  org.apache.hadoop.hdfs.server.namenode.TestFsck
                  org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
                  org.apache.hadoop.hdfs.TestListFilesInFileContext
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
                  org.apache.hadoop.hdfs.TestMultiThreadedHflush
                  org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
                  org.apache.hadoop.hdfs.TestDataTransferKeepalive
                  org.apache.hadoop.hdfs.TestDatanodeConfig
                  org.apache.hadoop.hdfs.TestRenameWhileOpen
                  org.apache.hadoop.hdfs.TestClientReportBadBlock
                  org.apache.hadoop.hdfs.TestSmallBlock
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
                  org.apache.hadoop.hdfs.TestHFlush
                  org.apache.hadoop.hdfs.TestParallelUnixDomainRead
                  org.apache.hadoop.hdfs.TestGetFileChecksum
                  org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
                  org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
                  org.apache.hadoop.hdfs.TestFileCreationEmpty
                  org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
                  org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
                  org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
                  org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
                  org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams
                  org.apache.hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
                  org.apache.hadoop.hdfs.TestDFSStartupVersions
                  org.apache.hadoop.hdfs.TestFileCreation
                  org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
                  org.apache.hadoop.hdfs.TestMissingBlocksAlert
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
                  org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
                  org.apache.hadoop.hdfs.server.namenode.TestStartup
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeRecovery
                  org.apache.hadoop.hdfs.TestRead
                  org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover
                  org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
                  org.apache.hadoop.hdfs.TestLocalDFS
                  org.apache.hadoop.hdfs.TestDatanodeDeath
                  org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
                  org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
                  org.apache.hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
                  org.apache.hadoop.hdfs.TestBlockReaderFactory
                  org.apache.hadoop.hdfs.TestDFSShell
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
                  org.apache.hadoop.hdfs.TestFetchImage
                  org.apache.hadoop.hdfs.TestListFilesInDFS
                  org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
                  org.apache.hadoop.hdfs.TestLeaseRecovery2
                  org.apache.hadoop.hdfs.TestFileCorruption
                  org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput
                  org.apache.hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler
                  org.apache.hadoop.hdfs.TestHDFSTrash
                  org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
                  org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
                  org.apache.hadoop.hdfs.server.namenode.TestMetaSave
                  org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
                  org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
                  org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations
                  org.apache.hadoop.hdfs.TestDistributedFileSystem
                  org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend
                  org.apache.hadoop.hdfs.tools.TestGetGroups
                  org.apache.hadoop.hdfs.server.namenode.TestBackupNode
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
                  org.apache.hadoop.hdfs.TestLargeBlock
                  org.apache.hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
                  org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
                  org.apache.hadoop.hdfs.server.namenode.TestAddBlock
                  org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
                  org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
                  org.apache.hadoop.hdfs.TestLease
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
                  org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
                  org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
                  org.apache.hadoop.hdfs.TestBlockReaderLocal
                  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
                  org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
                  org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
                  org.apache.hadoop.hdfs.TestLeaseRecovery
                  org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7711//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7711//console

This message is automatically generated.

> Add log category for long-running DFSClient notices
> ---------------------------------------------------
>
>                 Key: HDFS-4486
>                 URL: https://issues.apache.org/jira/browse/HDFS-4486
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Todd Lipcon
>            Assignee: Zhe Zhang
>            Priority: Minor
>         Attachments: HDFS-4486-20140820.patch, HDFS-4486-20140821.patch, hdfs-4486-20140821-2.patch
>
>
> There are a number of features in the DFS client which are transparent but can make a
fairly big difference for performance -- two in particular are short circuit reads and native
checksumming. Because we don't want log spew for clients like "hadoop fs -cat" we currently
log only at DEBUG level when these features are disabled. This makes it difficult to troubleshoot/verify
for long-running perf-sensitive clients like HBase.
> One simple solution is to add a new log category - eg o.a.h.h.DFSClient.PerformanceAdvisory
- which long-running clients could enable at DEBUG level without getting the full debug spew.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message