hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iñigo Goiri <elgo...@gmail.com>
Subject Re: Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
Date Thu, 15 Mar 2018 17:20:46 GMT
Thank you very much Allen for making the Windows build work again.
We are going through the unit tests and fixing them for Windows (as you
said mostly paths).
We got a couple related patches in already; this will help us track
progress.

On Thu, Mar 15, 2018 at 10:15 AM, Allen Wittenauer <aw@apache.org> wrote:

>
>         For my part of the HDFS bug bash, I’ve gotten the ASF Windows
> build working again. Starting tomorrow, results will be sent to the *-dev
> lists.
>
> A few notes:
>
> * It only runs the unit tests.  There’s not much point in running the
> other Yetus plugins since those are covered by the Linux one and this build
> is slow enough as it is.
>
> * There are two types of ASF build nodes: Windows Server 2012 and Windows
> Server 2016. This job can run on both and will use whichever one has a free
> slot.
>
> * It ALWAYS applies HADOOP-14667.05.patch prior to running.  As a result,
> this is only set up for trunk with no parameterization to run other
> branches.
>
> * The URI handling for file paths in hadoop-common and elsewhere is pretty
> broken on Windows, so many many many unit tests are failing and I wouldn't
> be surprised if Windows hadoop installs are horked as a result.
>
> * Runtime is about 12-13 hours with many tests taking significantly longer
> than their UNIX counterparts.  My guess is that this caused by winutils.
> Changing from winutils to Java 7 API calls would get this more in line and
> be a significant performance boost for Windows clients/servers as well.
>
> Have fun.
>
> =====
>
> For more details, see https://builds.apache.org/job/hadoop-trunk-win/406/
> <https://builds.apache.org/job/hadoop-trunk-win/406/>
>
> [Mar 14, 2018 6:26:58 PM] (xyao) HDFS-13251. Avoid using hard coded
> datanode data dirs in unit tests.
> [Mar 14, 2018 8:05:24 PM] (jlowe) MAPREDUCE-7064. Flaky test
> [Mar 14, 2018 8:14:36 PM] (inigoiri) HDFS-13198. RBF:
> RouterHeartbeatService throws out CachedStateStore
> [Mar 14, 2018 8:36:53 PM] (wangda) Revert "HADOOP-13707. If kerberos is
> enabled while HTTP SPNEGO is not
> [Mar 14, 2018 10:47:56 PM] (fabbri) HADOOP-15278 log s3a at info.
> Contributed by Steve Loughran.
>
>
>
>
> -1 overall
>
>
> The following subsystems voted -1:
>    unit
>
>
> The following subsystems are considered long running:
> (runtime bigger than 1h 00m 00s)
>    unit
>
>
> Specific tests:
>
>    Failed CTEST tests :
>
>       test_test_libhdfs_threaded_hdfs_static
>
>    Failed junit tests :
>
>       hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
>       hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
>       hadoop.fs.TestFsShellCopy
>       hadoop.fs.TestFsShellList
>       hadoop.fs.TestLocalFileSystem
>       hadoop.http.TestHttpServer
>       hadoop.http.TestHttpServerLogs
>       hadoop.io.compress.TestCodec
>       hadoop.io.nativeio.TestNativeIO
>       hadoop.ipc.TestSocketFactory
>       hadoop.metrics2.impl.TestStatsDMetrics
>       hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal
>       hadoop.security.TestSecurityUtil
>       hadoop.security.TestShellBasedUnixGroupsMapping
>       hadoop.security.token.TestDtUtilShell
>       hadoop.util.TestNativeCodeLoader
>       hadoop.fs.TestWebHdfsFileContextMainOperations
>       hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy
>       hadoop.hdfs.crypto.TestHdfsCryptoStreams
>       hadoop.hdfs.qjournal.client.TestQuorumJournalManager
>       hadoop.hdfs.qjournal.server.TestJournalNode
>       hadoop.hdfs.qjournal.server.TestJournalNodeSync
>       hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
>       hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
>       hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
>       hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
>       hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
>       hadoop.hdfs.server.datanode.fsdataset.impl.
> TestLazyPersistLockedMemory
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
>       hadoop.hdfs.server.datanode.fsdataset.impl.
> TestLazyPersistReplicaPlacement
>       hadoop.hdfs.server.datanode.fsdataset.impl.
> TestLazyPersistReplicaRecovery
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
>       hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
>       hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage
>       hadoop.hdfs.server.datanode.TestBlockRecovery
>       hadoop.hdfs.server.datanode.TestBlockScanner
>       hadoop.hdfs.server.datanode.TestDataNodeFaultInjector
>       hadoop.hdfs.server.datanode.TestDataNodeMetrics
>       hadoop.hdfs.server.datanode.TestDataNodeUUID
>       hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
>       hadoop.hdfs.server.datanode.TestDirectoryScanner
>       hadoop.hdfs.server.datanode.TestHSync
>       hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame
>       hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand
>       hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC
>       hadoop.hdfs.server.federation.router.TestRouterAdminCLI
>       hadoop.hdfs.server.mover.TestStorageMover
>       hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
>       hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
>       hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
>       hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstruction
> WithSnapshot
>       hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
>       hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
>       hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff
>       hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
>       hadoop.hdfs.server.namenode.TestAddBlock
>       hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands
>       hadoop.hdfs.server.namenode.TestCheckpoint
>       hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
>       hadoop.hdfs.server.namenode.TestEditLogRace
>       hadoop.hdfs.server.namenode.TestFileTruncate
>       hadoop.hdfs.server.namenode.TestFsck
>       hadoop.hdfs.server.namenode.TestFSImage
>       hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
>       hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
>       hadoop.hdfs.server.namenode.TestNameNodeMXBean
>       hadoop.hdfs.server.namenode.TestNestedEncryptionZones
>       hadoop.hdfs.server.namenode.TestQuotaByStorageType
>       hadoop.hdfs.server.namenode.TestStartup
>       hadoop.hdfs.TestDatanodeRegistration
>       hadoop.hdfs.TestDecommission
>       hadoop.hdfs.TestDFSOutputStream
>       hadoop.hdfs.TestDFSShell
>       hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
>       hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
>       hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
>       hadoop.hdfs.TestFetchImage
>       hadoop.hdfs.TestFileAppend
>       hadoop.hdfs.TestFileConcurrentReader
>       hadoop.hdfs.TestFileCreation
>       hadoop.hdfs.TestFileLengthOnClusterRestart
>       hadoop.hdfs.TestHDFSFileSystemContract
>       hadoop.hdfs.TestHDFSServerPorts
>       hadoop.hdfs.TestHDFSTrash
>       hadoop.hdfs.TestHFlush
>       hadoop.hdfs.TestLeaseRecovery
>       hadoop.hdfs.TestLocalDFS
>       hadoop.hdfs.TestMaintenanceState
>       hadoop.hdfs.TestMiniDFSCluster
>       hadoop.hdfs.TestPread
>       hadoop.hdfs.TestQuota
>       hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData
>       hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData
>       hadoop.hdfs.TestReadStripedFileWithMissingBlocks
>       hadoop.hdfs.TestReconstructStripedFile
>       hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
>       hadoop.hdfs.TestSecureEncryptionZoneWithKMS
>       hadoop.hdfs.TestTrashWithSecureEncryptionZones
>       hadoop.hdfs.tools.TestDebugAdmin
>       hadoop.hdfs.tools.TestDFSAdmin
>       hadoop.hdfs.tools.TestDFSAdminWithHA
>       hadoop.hdfs.tools.TestGetConf
>       hadoop.hdfs.web.TestWebHDFS
>       hadoop.hdfs.web.TestWebHdfsUrl
>       hadoop.fs.http.server.TestHttpFSServerWebServer
>       hadoop.hdfs.nfs.nfs3.TestRpcProgramNfs3
>       hadoop.hdfs.nfs.nfs3.TestWrites
>       hadoop.yarn.logaggregation.filecontroller.ifile.
> TestLogAggregationIndexFileController
>       hadoop.yarn.logaggregation.TestAggregatedLogFormat
>       hadoop.yarn.server.nodemanager.containermanager.
> launcher.TestContainerLaunch
>       hadoop.yarn.server.nodemanager.containermanager.linux.privileged.
> TestPrivilegedOperationExecutor
>       hadoop.yarn.server.nodemanager.containermanager.linux.resources.
> TestCGroupsHandlerImpl
>       hadoop.yarn.server.nodemanager.containermanager.linux.runtime.
> TestDockerContainerRuntime
>       hadoop.yarn.server.nodemanager.containermanager.linux.runtime.
> TestJavaSandboxLinuxContainerRuntime
>       hadoop.yarn.server.nodemanager.containermanager.logaggregation.
> TestAppLogAggregatorImpl
>       hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
>       hadoop.yarn.server.nodemanager.recovery.
> TestNMLeveldbStateStoreService
>       hadoop.yarn.server.nodemanager.TestContainerExecutor
>       hadoop.yarn.server.nodemanager.TestLocalDirsHandlerService
>       hadoop.yarn.server.nodemanager.TestNodeManagerResync
>       hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage
>       hadoop.yarn.server.applicationhistoryservice.
> TestApplicationHistoryServer
>       hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilt
> erForV1
>       hadoop.yarn.server.resourcemanager.metrics.
> TestSystemMetricsPublisher
>       hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore
>       hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.
> TestLeveldbConfigurationStore
>       hadoop.yarn.server.resourcemanager.scheduler.
> capacity.TestCapacityScheduler
>       hadoop.yarn.server.resourcemanager.scheduler.constraint.
> TestPlacementProcessor
>       hadoop.yarn.server.resourcemanager.scheduler.fair.
> TestAllocationFileLoaderService
>       hadoop.yarn.server.resourcemanager.TestResourceTrackerService
>       hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore
>       hadoop.yarn.server.timelineservice.reader.
> TestTimelineReaderWebServicesHBaseStorage
>       hadoop.yarn.server.timelineservice.storage.flow.
> TestHBaseStorageFlowActivity
>       hadoop.yarn.server.timelineservice.storage.flow.
> TestHBaseStorageFlowRun
>       hadoop.yarn.server.timelineservice.storage.flow.
> TestHBaseStorageFlowRunCompaction
>       hadoop.yarn.server.timelineservice.storage.
> TestHBaseTimelineStorageApps
>       hadoop.yarn.server.timelineservice.storage.
> TestHBaseTimelineStorageEntities
>       hadoop.yarn.server.timelineservice.storage.
> TestHBaseTimelineStorageSchema
>       hadoop.yarn.applications.distributedshell.TestDistributedShell
>       hadoop.mapred.TestTaskProgressReporter
>       hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter
>       hadoop.mapreduce.lib.output.TestFileOutputCommitter
>       hadoop.mapred.TestJavaSerialization
>       hadoop.mapred.TestLocalJobSubmission
>       hadoop.mapred.TestMiniMRChildTask
>       hadoop.mapred.TestMRTimelineEventHandling
>       hadoop.mapreduce.v2.TestMRJobs
>       hadoop.mapreduce.v2.TestUberAM
>       hadoop.yarn.service.client.TestBuildExternalComponents
>       hadoop.yarn.service.client.TestServiceCLI
>       hadoop.yarn.service.monitor.TestServiceMonitor
>       hadoop.yarn.service.providers.TestAbstractClientProvider
>       hadoop.yarn.service.TestServiceAM
>       hadoop.yarn.service.TestYarnNativeServices
>       hadoop.mapred.nativetask.handlers.TestNativeCollectorOnlyHandler
>       hadoop.mapred.uploader.TestFrameworkUploader
>       hadoop.tools.TestDistCpSystem
>       hadoop.tools.TestHadoopArchiveLogs
>       hadoop.mapred.gridmix.TestGridMixClasses
>       hadoop.fs.azure.TestClientThrottlingAnalyzer
>       hadoop.yarn.sls.TestReservationSystemInvariants
>
>
>   CTEST:
>
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt>
> [28K]
>
>   unit:
>
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-common-project_hadoop-common.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt>  [276K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt>  [1.5M]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt>
> [68K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt>  [20K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-nfs.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-nfs.txt>  [16K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-common.txt>  [40K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-server_hadoop-yarn-server-nodemanager.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-
> hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-
> nodemanager.txt>  [216K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-server_hadoop-yarn-server-applicationhistoryservice.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt>
> [32K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-server_hadoop-yarn-server-resourcemanager.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt>  [116K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt>
> [12K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt>
> [12K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-applications_hadoop-yarn-applications-distributedshell.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-applications_hadoop-yarn-applications-
> distributedshell.txt>  [20K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-
> mapreduce-client-core.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_
> hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt> [136K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-
> mapreduce-client-jobclient.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_
> hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt>  [108K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-
> yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-yarn-project_hadoop-
> yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-
> yarn-services-core.txt>  [92K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-
> mapreduce-client-nativetask.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_
> hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt>  [12K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-
> mapreduce-client-uploader.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_
> hadoop-mapreduce-client_hadoop-mapreduce-client-uploader.txt>  [20K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-tools_hadoop-distcp.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-
> hadoop-tools_hadoop-distcp.txt>  [16K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-tools_hadoop-archive-logs.txt <
> https://builds.apache.org/job/hadoop-trunk-win/406/
> artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt>  [8.0K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-tools_hadoop-gridmix.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-
> hadoop-tools_hadoop-gridmix.txt>  [12K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-tools_hadoop-azure.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-
> hadoop-tools_hadoop-azure.txt>  [16K]
>       https://builds.apache.org/job/hadoop-trunk-win/406/artifact/
> out/patch-unit-hadoop-tools_hadoop-sls.txt <https://builds.apache.org/
> job/hadoop-trunk-win/406/artifact/out/patch-unit-
> hadoop-tools_hadoop-sls.txt>  [12K]
>
> Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org <
> http://yetus.apache.org/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message