hadoop-ozone-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sammic...@apache.org
Subject [hadoop-ozone] branch HDDS-1564 updated (1938862 -> 40da915)
Date Mon, 23 Dec 2019 03:19:58 GMT
This is an automated email from the ASF dual-hosted git repository.

sammichen pushed a change to branch HDDS-1564
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.


    from 1938862  HDDS-1576. Support configure more than one raft log storage to host m…
(#117)
     new 9ebcb54  HDDS-2505. Fix logic related to SCM address calculation in HddsUtils
     new 431c85d  HDDS-2444. Remove server side dependencies from ozonefs jar files
     new 28dc284  HDDS-2619. Fix Sonar issues in ReconTaskControllerImpl
     new 2defafa  HDDS-2655. Use pre-compiled Pattern in NetUtils#normalize
     new 367846d  HDDS-2658. Insight log level reset does not work
     new a84a9f4  HDDS-2654. Conditionally enable profiling at the kernel level
     new f831950  HDDS-2639. TestTableCacheImpl is flaky (#278)
     new ab740dc  HDDS-2657. Key get command creates the output file even in case of KEY_NOT_FOUND
(#290)
     new bb22b3c  HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader (#137)
     new 27c384f  HDDS-2573. Handle InterruptedException in KeyOutputStream. (#257)
     new cdb9e15  HDDS-2637. Handle LeaderNot ready exception in OzoneManager StateMachine
and upgrade ratis to latest version. (#276)
     new 8c07266  HDDS-2386. Implement incremental ChunkBuffer. (#205)
     new 4efd8aa  HDDS-2620. Fix listMultipartupload API. (#263)
     new bf9cbf1  HDDS-1722. Use the bindings in ReconSchemaGenerationModule to create Recon
SQL tables on startup. (#294)
     new 13c1db0  HDDS-2571. Handle InterruptedException in SCMPipelineManager (#295)
     new 0543da7  HDDS-2656. Prefer execute() over submit() if the returned Future is ignored
(#302)
     new 5c21608  HDDS-2475. Unregister ContainerMetadataScrubberMetrics on thread exit (#286)
     new f0ea1a4  HDDS-2653. Improve executor memory usage in new Freon tests (#284)
     new 8b7d7ef  HDDS-2608. Provide command to wait until SCM is out from the safe-mode (#250)
     new 3d1b44d  HDDS-2646. Start acceptance tests only if at least one THREE pipeline is
available (#282)
     new d903120  HDDS-2622. Add config parameter for setting limit on total bytes of pending
requests in Ratis. (#264)
     new d7a4ad4  HDDS-2667. Prometheus reports invalid metric type
     new 495700c  Addendum HDDS-2646. Start acceptance tests only if at least one THREE pipeline
is available (#282)
     new 0b5d319  HDDS-2506. Remove keyAllocationInfo and replication info from the auditLog
(#288)
     new bf36ab0  HDDS-2565. Handle InterruptedException in VolumeSet
     new 0319d42  HDDS-2600. Move chaos test to org.apache.hadoop.ozone.chaos package. (#307)
     new 74bab02  HDDS-1873. Add API to get last completed times for every Recon task. (#233)
     new f2965ac  HDDS-2666. Suppress loader constraint violation message in TestOzoneFileSystemWithMocks
(#303)
     new 30b9d58  HDDS-2530. Sonar : refactor verifyResourceName in HddsClientUtils to fix
Sonar errors (#293)
     new d89f6d4  HDDS-2629. Ozone CLI: CreationTime/modifyTime of volume/bucket/key in…
(#272)
     new 93abc66  HDDS-2677. Acceptance test may fail despite success status (#312)
     new 80975cc  HDDS-2570. Handle InterruptedException in ProfileServlet. (#308)
     new 7478168  HDDS-1987. Fix listStatus API (#108)
     new 4c046d4  HDDS-2636. Refresh pipeline information in OzoneManager lookupFile call.
(#301)
     new 6c5d0fb  HDDS-2678. Add thread name to log pattern (#315)
     new 7657fbd  HDDS-2662. Update gRPC and datanode protobuf version in Ozone
     new e306d51  HDDS-2681. Add leak detection memory flags to MiniOzoneChaosCluster
     new 0f8239f  HDDS-1991. Remove RatisClient in OM HA. (#319)
     new 421337b  HDDS-2692. Seek to file end throws EOF Exception. (#326)
     new b33b64d  HDDS-2689. OMException NOT_A_FILE missing space in the exception message.
(#327)
     new 1ecd057  HDDS-2687. Fix sonar issues in package org.apache.hadoop.ozone.recon.api.
(#325)
     new 6f743cc  HDDS-2668. Sonar : fix issues reported in BlockManagerImpl (#317)
     new b324d4a  HDDS-2645. Refactor MiniOzoneChaosCluster to a different package to add
filesystem tests. (#292)
     new 1b43a92  HDDS-2683. Ratis MaxBuffer should be the same size as the segment size.
(#320)
     new 91cab42  HDDS-1993. Merge OzoneManagerRequestHandler and OzoneManagerHARequest…
(#304)
     new d0ec961  HDDS-2693. HddsVolume mixes ChunkLayOutVersion and DataNodeLayoutVersion
(#330)
     new 554c311  HDDS-2105. Merge OzoneClientFactory#getRpcClient functions (#148)
     new 14f13c9  HDDS-2389. add toStateMachineLogEntryString provider in Ozone's ContainerStateMachine.
(#298)
     new 5a507b4  HDDS-2532. Sonar : fix issues in OzoneQuota (#311)
     new db20ab5  HDDS-2561. Handle InterruptedException in LeaseManager (#338)
     new 86af50f  HDDS-2704. Source tar file is not created during the relase build (#334)
     new 3da6188  HDDS-2711. OM does not report JVM metrics (#346)
     new 54deaa0  HDDS-2688. Refactor container response builders to hadoop-hdds/common (#324)
     new db7efc7  HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception
on container creation failure (#296).
     new 3799c3e  HDDS-2588. Consolidate compose environments
     new 5707737  HDDS-2684. Refactor common test utilities to hadoop-hdds/common (#322)
     new 551a585  HDDS-2242. Avoid unnecessary rpc needed to discover the pipeline leader.
(#313)
     new 71021e4  HDDS-2553. Sonar: Iterator.next() methods should throw NoSuchElementException
     new c5da69e  HDDS-2423. Add the recover-trash command client side code
     new f94f8f7  HDDS-2414. Simplify robot tests with removing output greps
     new 341785d  HDDS-2680. Fix updating lastAppliedIndex in OzoneManagerStateMachine. (#316)
     new c311c99  HDDS-2675. Sonar : fix issues in PipelineStateManager (#347)
     new 7b2355a  HDDS-2578. Handle InterruptedException in RandomKeyGenerator (#305)
     new 88727f7  HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager class. (#332)
     new 3b6eb17  HDDS-2729. Enable multilingual Hugo features in ozone docs (#357)
     new 7f9368b  HDDS-2751. Wrong number of placeholders in log message (#362)
     new eb341bd  HDDS-2723. Wrap InputStream with try-with-resource in TestOzoneFSInputStream.
(#361)
     new bdb0839  HDDS-2724. Sonar : Fix sonar issues in package 'org.apache.hadoop.ozo…
(#354)
     new 4753d70  HDDS-2669. Sonar : fix issues in scm Pipeline (#360)
     new 3ecb968  HDDS-2752. Unnecessary calls to isNoneEmpty and isAllEmpty (#364)
     new c7caff1  HDDS-2630. NullPointerException in S3g. (#336)
     new 8158b12  Revert "HDDS-2704. Source tar file is not created during the relase build
(#334)" (#368)
     new 552f0b3  HDDS-2623. Expose SCMDatanodeProtocolServer RPC endpoint through Recon.
(#353)
     new bd14928  HDDS-2070. Create insight point to debug one specific pipeline (#299)
     new 27a77e9  HDDS-2722. Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
(#352)
     new 3a6fa66  HDDS-2703. OzoneFSInputStream to support ByteBufferReadable (#345)
     new 1331aa1  HDDS-2739. No need to try install awscli before each test (#359)
     new a359ab6  HDDS-2652. Remove unnecessary log statement. (#371)
     new 21281c6  HDDS-2709. Maven property skipShade should not skip ozonefs compilation
(#342)
     new d318ee2  HDDS-2735. Let GitHub Actions run acceptance check in parallel (#358)
     new 79ea2d9  HDDS-2774. Hadoop 3.1 acceptance test fails with apk command not found (#374)
     new f472e8d  HDDS-2776. NPE when stop recon while start recon failed (#376)
     new 0d91171  HDDS-2762. Sonar warnings fixed (#373)
     new 57d3fe5  HDDS-2755. Compare transactionID and updateID of Volume operations to avoid
replaying transactions (#369)
     new f99848d  HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine.
(#351)
     new 7a6b6a1  HDDS-2685. Fix Rename API in BasicOzoneFileSystem (#321)
     new a96ff61  HDDS-2704. Source tar file is not created during the relase build. (#380)
     new d5034a0  HDDS-2777. Add bytes read statistics to Ozone FileSystem implementation
(#382)
     new 51b8cd1  HDDS-2786. ITestOzoneContractSeek zero byte file failures (#384)
     new 417c958  Revert "Rebase Fix"
     new 40da915  HDDS-1564. Rebase fix.

The 91 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/workflows/post-commit.yml                  |   6 -
 .github/workflows/pr.yml                           |   6 -
 .../hadoop/hdds/scm/client/HddsClientUtils.java    |  75 +--
 .../hadoop/hdds/scm/storage/BlockOutputStream.java |   3 +-
 .../apache/hadoop/hdds/scm/storage/BufferPool.java |   4 +-
 hadoop-hdds/common/pom.xml                         |   9 +-
 .../java/org/apache/hadoop/hdds/HddsUtils.java     | 216 ++++----
 .../org/apache/hadoop/hdds/client/OzoneQuota.java  |  12 +-
 .../org/apache/hadoop/hdds/ratis/RatisHelper.java  |  13 +-
 .../ReconConfigKeys.java}                          |  27 +-
 .../apache/hadoop/hdds/recon}/package-info.java    |   4 +-
 .../hadoop/hdds/scm/ByteStringConversion.java      |  16 +-
 .../org/apache/hadoop/hdds/scm/ScmConfigKeys.java  |   5 +
 .../org/apache/hadoop/hdds/scm/net/NetUtils.java   |  19 +-
 .../apache/hadoop/hdds/scm/pipeline/Pipeline.java  |  25 +-
 .../ContainerCommandResponseBuilders.java          | 269 +++++++++
 .../certificate/authority/DefaultCAServer.java     |   4 +-
 .../client/DefaultCertificateClient.java           |   2 +-
 .../hadoop/hdds/utils/RocksDBStoreMBean.java       |   2 +-
 .../apache/hadoop/hdds/utils/db/DBCheckpoint.java  |  12 +
 .../hadoop/hdds/utils/db/RocksDBCheckpoint.java    |  11 +
 .../hadoop/hdds/utils/db/cache/TableCacheImpl.java |   5 +-
 .../org/apache/hadoop/ozone/OzoneConfigKeys.java   |   5 +
 .../java/org/apache/hadoop/ozone/OzoneConsts.java  |   2 +
 .../apache/hadoop/ozone/common/ChunkBuffer.java    |  88 ++-
 .../common/ChunkBufferImplWithByteBuffer.java      |  38 +-
 .../ozone/common/IncrementalChunkBuffer.java       | 287 ++++++++++
 .../ozone/container/common/helpers/BlockData.java  |   5 -
 .../apache/hadoop/ozone/lease/LeaseManager.java    |   4 +-
 .../apache/hadoop/ozone/web/utils/JsonUtils.java   |  15 +-
 .../common/src/main/resources/ozone-default.xml    |  38 +-
 .../hadoop/hdds/conf/SimpleConfiguration.java      |   2 +-
 .../hadoop/hdds/conf/TestOzoneConfiguration.java   |  10 +-
 .../hadoop/hdds/protocol/MockDatanodeDetails.java  | 125 +++++
 .../apache/hadoop/hdds/scm/net/TestNetUtils.java   |  30 +-
 .../hadoop/hdds/scm/pipeline/MockPipeline.java     |  74 +++
 .../hadoop/hdds/utils/MockGatheringChannel.java    |  74 +++
 .../hdds/utils/db/cache/TestTableCacheImpl.java    |  62 +--
 .../hadoop/ozone/audit/TestOzoneAuditLogger.java   |  10 +-
 .../apache/hadoop/ozone/audit/package-info.java    |   2 +-
 .../hadoop/ozone/common/TestChunkBuffer.java       |  47 +-
 .../{log4j2.properties => auditlog.properties}     |   0
 hadoop-hdds/container-service/pom.xml              |   6 +-
 .../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 139 ++---
 .../container/common/helpers/ContainerUtils.java   |  85 +--
 .../container/common/impl/HddsDispatcher.java      |   7 +-
 .../common/statemachine/EndpointStateMachine.java  |   2 +-
 .../commandhandler/CommandDispatcher.java          |   2 +-
 .../server/ratis/ContainerStateMachine.java        |  30 +-
 .../transport/server/ratis/XceiverServerRatis.java |  42 +-
 .../ozone/container/common/volume/HddsVolume.java  |   3 +-
 .../container/common/volume/HddsVolumeChecker.java |   2 +-
 .../ozone/container/common/volume/VolumeSet.java   |   1 +
 .../ozone/container/keyvalue/KeyValueHandler.java  | 149 ++---
 .../container/keyvalue/helpers/BlockUtils.java     |  88 +--
 .../container/keyvalue/helpers/ChunkUtils.java     |  26 +-
 .../keyvalue/helpers/KeyValueContainerUtil.java    |  27 -
 .../container/keyvalue/helpers/SmallFileUtils.java |  91 ---
 .../container/keyvalue/impl/BlockManagerImpl.java  |  65 +--
 .../keyvalue/impl/ChunkManagerDummyImpl.java       |   7 +-
 .../container/keyvalue/impl/ChunkManagerImpl.java  |  13 +-
 .../keyvalue/interfaces/ChunkManager.java          |  13 +-
 .../container/ozoneimpl/ContainerController.java   |   9 +
 .../ozoneimpl/ContainerDataScrubberMetrics.java    |   8 +-
 .../ozoneimpl/ContainerMetadataScanner.java        |  18 +-
 .../ContainerMetadataScrubberMetrics.java          |   8 +-
 .../replication/ReplicationSupervisor.java         |   2 +-
 .../ozone/container/ContainerTestHelper.java       |  87 +--
 .../common/impl/TestContainerPersistence.java      |  33 +-
 .../container/common/impl/TestHddsDispatcher.java  |   6 +-
 .../container/common/volume/TestVolumeSet.java     |  24 +
 .../container/keyvalue/TestChunkManagerImpl.java   |  18 +-
 .../container/keyvalue/helpers/TestChunkUtils.java |  13 +-
 .../ozoneimpl/TestContainerScrubberMetrics.java    |  35 +-
 hadoop-hdds/docs/config.yaml                       |   9 +-
 hadoop-hdds/docs/content/_index.md                 |   8 +-
 .../default.md => content/_index.zh.md}            |   6 +-
 .../default.md => content/concept/Datanodes.zh.md} |   8 +-
 .../default.md => content/concept/_index.zh.md}    |   8 +-
 .../themes/ozonedoc/layouts/_default/section.html  |   4 +
 .../themes/ozonedoc/layouts/_default/single.html   |   8 +-
 .../docs/themes/ozonedoc/layouts/index.html        |   4 +-
 .../ozonedoc/layouts/partials/languages.html}      |  10 +-
 .../apache/hadoop/hdds/server/BaseHttpServer.java  |  19 +-
 .../apache/hadoop/hdds/server/ProfileServlet.java  |   7 +-
 .../hadoop/hdds/server/PrometheusMetricsSink.java  |   5 +-
 .../org/apache/hadoop/hdds/server/ServerUtils.java |   8 +
 .../hdds/server/TestPrometheusMetricsSink.java     |   2 +-
 hadoop-hdds/pom.xml                                |   7 +
 .../hdds/scm/container/SCMContainerManager.java    |  48 +-
 .../scm/container/states/ContainerStateMap.java    |   2 +-
 .../scm/pipeline/BackgroundPipelineCreator.java    |   2 +-
 .../hdds/scm/pipeline/PipelineActionHandler.java   |   2 +-
 .../hdds/scm/pipeline/PipelineStateManager.java    |   7 +-
 .../hdds/scm/pipeline/SCMPipelineManager.java      |   3 +-
 .../scm/safemode/HealthyPipelineSafeModeRule.java  |   2 +-
 .../scm/server/OzoneStorageContainerManager.java   |  30 +-
 .../hdds/scm/server/SCMDatanodeProtocolServer.java |  12 +-
 .../hdds/scm/server/SCMSecurityProtocolServer.java |   2 +-
 .../hdds/scm/server/StorageContainerManager.java   |   9 +-
 .../org/apache/hadoop/hdds/scm/HddsTestUtils.java  |   5 +-
 .../java/org/apache/hadoop/hdds/scm/TestUtils.java |  74 +--
 .../command/TestCommandStatusReportHandler.java    |   3 +-
 .../hadoop/hdds/scm/container/MockNodeManager.java |   4 +-
 .../container/TestCloseContainerEventHandler.java  |   4 +-
 .../scm/container/TestContainerActionsHandler.java |   4 +-
 .../scm/container/TestContainerStateManager.java   |  12 +-
 .../TestIncrementalContainerReportHandler.java     |  20 +-
 .../hdds/scm/container/TestReplicationManager.java |   4 +-
 .../scm/container/TestSCMContainerManager.java     |   5 +-
 .../algorithms/TestContainerPlacementFactory.java  |   4 +-
 .../TestSCMContainerPlacementCapacity.java         |   4 +-
 .../TestSCMContainerPlacementRackAware.java        |   6 +-
 .../TestSCMContainerPlacementRandom.java           |   4 +-
 .../hdds/scm/node/TestContainerPlacement.java      |   3 +-
 .../hadoop/hdds/scm/node/TestDeadNodeHandler.java  |  19 +-
 .../hdds/scm/node/TestNodeReportHandler.java       |   3 +-
 .../hadoop/hdds/scm/node/TestSCMNodeManager.java   |  10 +-
 .../hadoop/hdds/scm/node/TestStatisticsUpdate.java |   5 +-
 .../TestPipelineDatanodesIntersection.java         |   2 +-
 .../scm/pipeline/TestPipelinePlacementPolicy.java  |   2 +-
 .../scm/pipeline/TestPipelineStateManager.java     |   9 +-
 .../scm/pipeline/TestRatisPipelineProvider.java    |  10 +-
 .../scm/pipeline/TestSimplePipelineProvider.java   |   7 +-
 .../scm/server/TestSCMBlockProtocolServer.java     |   4 +-
 .../server/TestSCMDatanodeHeartbeatDispatcher.java |   8 +-
 .../ozone/container/common/TestEndPoint.java       |  21 +-
 .../hadoop/hdds/scm/cli/SafeModeCommands.java      |   1 +
 .../hdds/scm/cli/SafeModeWaitSubcommand.java       |  93 ++++
 .../apache/hadoop/ozone/client/OzoneBucket.java    |  11 +-
 .../hadoop/ozone/client/OzoneClientFactory.java    |  68 +--
 .../org/apache/hadoop/ozone/client/OzoneKey.java   |  14 +-
 .../apache/hadoop/ozone/client/OzoneVolume.java    |   9 +-
 .../ozone/client/io/BlockOutputStreamEntry.java    |   9 +-
 .../client/io/BlockOutputStreamEntryPool.java      |  51 +-
 .../hadoop/ozone/client/io/KeyInputStream.java     |  12 +-
 .../hadoop/ozone/client/io/KeyOutputStream.java    | 153 +++---
 .../ozone/client/protocol/ClientProtocol.java      |  13 +
 .../apache/hadoop/ozone/client/rpc/RpcClient.java  |   8 +
 .../hadoop/ozone/client/TestHddsClientUtils.java   |  46 ++
 hadoop-ozone/common/pom.xml                        |   8 -
 .../main/java/org/apache/hadoop/ozone/OmUtils.java |  62 +--
 .../org/apache/hadoop/ozone/om/OMConfigKeys.java   |   9 -
 .../apache/hadoop/ozone/om/OMMetadataManager.java  |   3 +-
 .../hadoop/ozone/om/exceptions/OMException.java    |   4 +-
 .../om/exceptions/OMLeaderNotReadyException.java   |  22 +-
 ...derException.java => OMNotLeaderException.java} |  32 +-
 .../ozone/om/ha/OMFailoverProxyProvider.java       |   2 +-
 .../hadoop/ozone/om/helpers/OMRatisHelper.java     |  43 --
 .../apache/hadoop/ozone/om/helpers/OmKeyArgs.java  |   2 -
 .../ozone/om/protocol/OzoneManagerHAProtocol.java  |   6 +-
 .../ozone/om/protocol/OzoneManagerProtocol.java    |  15 +-
 ...OzoneManagerProtocolClientSideTranslatorPB.java |  90 ++-
 .../src/main/proto/OzoneManagerProtocol.proto      |  22 +
 .../java/org/apache/hadoop/ozone/TestOmUtils.java  |  65 +--
 hadoop-ozone/csi/pom.xml                           |  11 +-
 hadoop-ozone/dev-support/checks/unit.sh            |   2 +-
 hadoop-ozone/dist/pom.xml                          |  39 ++
 .../dist/src/main/compose/ozone-mr/common-config   |   1 +
 .../compose/ozone-mr/hadoop27/docker-compose.yaml  |   1 +
 .../compose/ozone-mr/hadoop31/docker-compose.yaml  |   1 +
 .../src/main/compose/ozone-mr/hadoop31/test.sh     |   8 +-
 .../compose/ozone-mr/hadoop32/docker-compose.yaml  |   1 +
 .../compose/ozone-om-ha-s3/docker-compose.yaml     |   1 +
 .../dist/src/main/compose/ozone-recon/.env         |  18 -
 .../main/compose/ozone-recon/docker-compose.yaml   |  65 ---
 .../src/main/compose/ozone-recon/docker-config     |  35 --
 .../compose/ozone-topology/docker-compose.yaml     |   1 +
 .../src/main/compose/ozone-topology/docker-config  |   1 +
 hadoop-ozone/dist/src/main/compose/ozone/README.md |  99 ++++
 .../src/main/compose/ozone/docker-compose.yaml     |  91 +--
 .../dist/src/main/compose/ozone/docker-config      |  11 +-
 .../compose/{ozoneperf => ozone}/freon-ockg.yaml   |  19 +-
 .../compose/{ozoneperf => ozone}/freon-rk.yaml     |  19 +-
 .../docker-config => ozone/monitoring.conf}        |  15 +-
 .../dist/src/main/compose/ozone/monitoring.yaml    |  56 ++
 .../{ozone-recon/test.sh => ozone/profiling.yaml}  |  30 +-
 .../compose/{ozoneperf => ozone}/prometheus.yml    |  20 +-
 .../{ozone-om-ha/startOM.sh => ozone/run.sh}       |  13 +-
 hadoop-ozone/dist/src/main/compose/ozone/test.sh   |   9 +-
 .../src/main/compose/ozoneblockade/docker-config   |   2 +
 hadoop-ozone/dist/src/main/compose/ozoneperf/.env  |  18 -
 .../dist/src/main/compose/ozoneperf/README.md      |  74 ---
 .../src/main/compose/ozoneperf/docker-compose.yaml |  84 ---
 .../dist/src/main/compose/ozoneperf/docker-config  |  36 --
 .../dist/src/main/compose/ozoneperf/test.sh        |  30 -
 .../compose/ozones3-haproxy/docker-compose.yaml    |   1 +
 hadoop-ozone/dist/src/main/compose/ozones3/.env    |  18 -
 .../src/main/compose/ozones3/docker-compose.yaml   |  58 --
 hadoop-ozone/dist/src/main/compose/ozones3/test.sh |  34 --
 .../src/main/compose/ozonescripts/docker-config    |   3 +-
 .../compose/ozonesecure-mr/docker-compose.yaml     |   1 +
 .../main/compose/ozonesecure/docker-compose.yaml   |   1 +
 hadoop-ozone/dist/src/main/compose/testlib.sh      |  53 +-
 .../dist/src/main/dockerlibexec/entrypoint.sh      |   8 +
 .../dist/src/main/smoketest/basic/basic.robot      |   4 +-
 .../src/main/smoketest/basic/ozone-shell.robot     |  18 +-
 .../dist/src/main/smoketest/gdpr/gdpr.robot        |  16 +-
 .../dist/src/main/smoketest/omha/testOMHA.robot    |   4 +-
 .../dist/src/main/smoketest/ozonefs/ozonefs.robot  |  18 +-
 .../src/main/smoketest/s3/MultipartUpload.robot    |   2 +-
 .../dist/src/main/smoketest/s3/__init__.robot      |   2 +-
 .../dist/src/main/smoketest/s3/bucketcreate.robot  |   2 +-
 .../dist/src/main/smoketest/s3/buckethead.robot    |   2 +-
 .../dist/src/main/smoketest/s3/bucketlist.robot    |   2 +-
 .../dist/src/main/smoketest/s3/objectcopy.robot    |   2 +-
 .../dist/src/main/smoketest/s3/objectdelete.robot  |   2 +-
 .../src/main/smoketest/s3/objectmultidelete.robot  |   2 +-
 .../dist/src/main/smoketest/s3/objectputget.robot  |   2 +-
 hadoop-ozone/dist/src/shell/conf/log4j.properties  |  10 +-
 .../fault-injection-test/mini-chaos-tests}/pom.xml |  37 +-
 .../mini-chaos-tests/src/test/bin/start-chaos.sh   |  46 ++
 .../apache/hadoop/ozone/MiniOzoneChaosCluster.java |   6 +-
 .../hadoop/ozone/MiniOzoneLoadGenerator.java       | 107 +---
 .../hadoop/ozone/TestMiniChaosOzoneCluster.java    |  12 +-
 .../org/apache/hadoop/ozone/utils/LoadBucket.java  |  99 ++++
 .../hadoop/ozone/utils}/TestProbability.java       |   2 +-
 .../services/org.apache.hadoop.fs.FileSystem       |   0
 .../src/test/resources/log4j.properties            |   8 +-
 hadoop-ozone/fault-injection-test/pom.xml          |   1 +
 .../hadoop/ozone/insight/BaseInsightPoint.java     |   5 +-
 .../ozone/insight/BaseInsightSubCommand.java       |   2 +
 .../apache/hadoop/ozone/insight/InsightPoint.java  |   3 +-
 .../apache/hadoop/ozone/insight/LogSubcommand.java |  33 +-
 .../ozone/insight/datanode/RatisInsight.java       |  44 +-
 .../hadoop/ozone/insight/om/KeyManagerInsight.java |   4 +-
 .../hadoop/ozone/insight/om/OmProtocolInsight.java |   3 +-
 .../ozone/insight/scm/EventQueueInsight.java       |   4 +-
 .../ozone/insight/scm/NodeManagerInsight.java      |   4 +-
 .../ozone/insight/scm/ReplicaManagerInsight.java   |   4 +-
 .../scm/ScmProtocolBlockLocationInsight.java       |   3 +-
 .../scm/ScmProtocolContainerLocationInsight.java   |   3 +-
 .../insight/scm/ScmProtocolDatanodeInsight.java    |   3 +-
 .../insight/scm/ScmProtocolSecurityInsight.java    |   3 +-
 .../integration-test/src/test/bin/start-chaos.sh   |  35 --
 .../hadoop/hdds/scm/TestRatisPipelineLeader.java   | 129 +++++
 .../scm/pipeline/TestRatisPipelineProvider.java    |  16 +-
 .../apache/hadoop/ozone/MiniOzoneClusterImpl.java  |  10 +-
 .../hadoop/ozone/MiniOzoneHAClusterImpl.java       |  24 +-
 .../apache/hadoop/ozone/TestMiniOzoneCluster.java  |  18 +-
 .../hadoop/ozone/TestMiniOzoneHACluster.java       | 112 ++++
 .../hadoop/ozone/client/rpc/TestCommitWatcher.java |   4 +-
 .../client/rpc/TestOzoneAtRestEncryption.java      |  14 +-
 .../client/rpc/TestOzoneRpcClientAbstract.java     |  90 ++-
 .../ozone/client/rpc/TestSecureOzoneRpcClient.java |   8 +-
 .../ozone/container/TestContainerReplication.java  |   3 +-
 .../transport/server/ratis/TestCSMMetrics.java     |   4 +-
 .../container/metrics/TestContainerMetrics.java    |   7 +-
 .../container/ozoneimpl/TestOzoneContainer.java    |  15 +-
 .../ozoneimpl/TestOzoneContainerRatis.java         |   4 +-
 .../ozoneimpl/TestOzoneContainerWithTLS.java       |   7 +-
 .../ozoneimpl/TestSecureOzoneContainer.java        |   7 +-
 .../container/server/TestContainerServer.java      |  11 +-
 .../server/TestSecureContainerServer.java          |   8 +-
 .../hadoop/ozone/dn/scrubber/TestDataScrubber.java |   8 +-
 .../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 235 ++++++++
 .../hadoop/ozone/om/TestOMRatisSnapshots.java      |  20 +-
 .../apache/hadoop/ozone/om/TestOzoneManagerHA.java |  34 +-
 .../hadoop/ozone/ozShell/TestOzoneShellHA.java     |  20 +-
 .../hadoop/ozone/scm/node/TestSCMNodeMetrics.java  |   5 +-
 hadoop-ozone/ozone-manager/pom.xml                 |   6 +-
 .../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 172 ++++--
 .../hadoop/ozone/om/OMDBCheckpointServlet.java     |  19 +-
 .../java/org/apache/hadoop/ozone/om/OMStorage.java |  17 +-
 .../hadoop/ozone/om/OmMetadataManagerImpl.java     |  35 +-
 .../org/apache/hadoop/ozone/om/OzoneManager.java   |  88 +--
 .../hadoop/ozone/om/ratis/OMRatisSnapshotInfo.java |  15 +-
 .../ozone/om/ratis/OzoneManagerDoubleBuffer.java   |   7 +-
 .../ozone/om/ratis/OzoneManagerRatisClient.java    | 210 -------
 .../ozone/om/ratis/OzoneManagerRatisServer.java    | 119 ++--
 .../ozone/om/ratis/OzoneManagerRatisSnapshot.java  |  10 +-
 .../ozone/om/ratis/OzoneManagerStateMachine.java   | 199 ++++++-
 .../om/ratis/utils/OzoneManagerRatisUtils.java     |  16 +-
 .../hadoop/ozone/om/request/OMClientRequest.java   |  23 +
 .../hadoop/ozone/om/request/RequestAuditor.java    |   3 -
 .../ozone/om/request/file/OMFileCreateRequest.java |   6 +-
 .../request/s3/bucket/S3BucketCreateRequest.java   |  18 +-
 .../om/request/volume/OMVolumeCreateRequest.java   |  26 +-
 .../om/request/volume/OMVolumeDeleteRequest.java   |  51 +-
 .../ozone/om/request/volume/OMVolumeRequest.java   |  20 +
 .../om/request/volume/OMVolumeSetOwnerRequest.java |  24 +-
 .../om/request/volume/OMVolumeSetQuotaRequest.java |  24 +-
 .../hadoop/ozone/om/response/OMClientResponse.java |  29 +-
 .../response/file/OMDirectoryCreateResponse.java   |  28 +-
 .../response/s3/bucket/S3BucketCreateResponse.java |   6 +-
 .../S3MultipartUploadCommitPartResponse.java       |  71 +--
 .../om/response/volume/OMVolumeAclOpResponse.java  |   2 -
 .../om/response/volume/OMVolumeCreateResponse.java |  38 +-
 .../om/response/volume/OMVolumeDeleteResponse.java |  42 +-
 .../response/volume/OMVolumeSetOwnerResponse.java  |  56 +-
 .../response/volume/OMVolumeSetQuotaResponse.java  |  26 +-
 .../om/snapshot/OzoneManagerSnapshotProvider.java  |  10 +
 .../protocolPB/OzoneManagerHARequestHandler.java   |  39 --
 .../OzoneManagerHARequestHandlerImpl.java          |  56 --
 ...OzoneManagerProtocolServerSideTranslatorPB.java |  27 +-
 .../protocolPB/OzoneManagerRequestHandler.java     | 607 +--------------------
 .../hadoop/ozone/protocolPB/RequestHandler.java    |  28 +-
 .../hadoop/ozone/web/ozShell/OzoneAddress.java     |   3 +-
 .../ozone/web/ozShell/keys/GetKeyHandler.java      |   4 +-
 .../ozone/om/ScmBlockLocationTestingClient.java    |   4 +-
 .../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 256 ++++++++-
 .../org/apache/hadoop/ozone/om/TestOMStorage.java  |  91 +++
 .../apache/hadoop/ozone/om/TestTrashService.java   | 125 +++++
 .../ozone/om/ratis/TestOMRatisSnapshotInfo.java    |   5 +-
 ...tOzoneManagerDoubleBufferWithDummyResponse.java |   2 +-
 ...TestOzoneManagerDoubleBufferWithOMResponse.java |   3 +-
 .../om/ratis/TestOzoneManagerRatisServer.java      |  29 +-
 .../om/ratis/TestOzoneManagerStateMachine.java     | 215 ++++++++
 .../ozone/om/request/TestOMRequestUtils.java       |  17 +
 .../request/volume/TestOMVolumeCreateRequest.java  |  51 +-
 .../request/volume/TestOMVolumeDeleteRequest.java  |  46 +-
 .../om/request/volume/TestOMVolumeRequest.java     |  30 +
 .../volume/TestOMVolumeSetOwnerRequest.java        |  31 ++
 .../volume/TestOMVolumeSetQuotaRequest.java        |  38 +-
 .../ozone/om/response/TestOMResponseUtils.java     |   2 +-
 .../volume/TestOMVolumeCreateResponse.java         |   7 +-
 .../volume/TestOMVolumeDeleteResponse.java         |  13 +-
 .../volume/TestOMVolumeSetOwnerResponse.java       |  11 +-
 .../volume/TestOMVolumeSetQuotaResponse.java       |   7 +-
 .../ozone/web/ozShell/TestObjectPrinter.java       |   6 +-
 hadoop-ozone/ozonefs-lib-current/pom.xml           |  15 -
 .../fs/ozone/BasicOzoneClientAdapterImpl.java      |   2 +-
 .../hadoop/fs/ozone/BasicOzoneFileSystem.java      |  36 +-
 .../apache/hadoop/fs/ozone/OzoneFSInputStream.java |  45 +-
 .../hadoop/fs/ozone/TestOzoneFSInputStream.java    |  72 ++-
 .../hadoop/fs/ozone/TestOzoneFileSystem.java       |  12 +
 .../fs/ozone/TestOzoneFileSystemWithMocks.java     |   2 +
 .../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java  |  48 +-
 .../hadoop/fs/ozone/TestOzoneFsRenameDir.java      |  16 +-
 .../hadoop/fs/ozone/TestReadWriteStatistics.java   | 452 +++++++++++++++
 hadoop-ozone/pom.xml                               |   2 +-
 hadoop-ozone/recon/pom.xml                         |   5 +-
 .../hadoop/ozone/recon/ReconControllerModule.java  |   6 +-
 .../hadoop/ozone/recon/ReconSchemaManager.java     |  55 ++
 .../org/apache/hadoop/ozone/recon/ReconServer.java |  47 +-
 .../ozone/recon/api/ContainerKeyService.java       | 112 ++--
 ...lizationService.java => TaskStatusService.java} |  39 +-
 .../hadoop/ozone/recon/api/UtilizationService.java |   5 -
 .../recon/recovery/ReconOMMetadataManager.java     |   6 +
 .../recon/recovery/ReconOmMetadataManagerImpl.java |  11 +
 .../recon/scm/ReconDatanodeProtocolServer.java     |  76 +++
 .../hadoop/ozone/recon/scm/ReconNodeManager.java   |  29 +-
 .../recon/scm/ReconStorageContainerManager.java    | 134 +++++
 .../recon/{recovery => scm}/package-info.java      |   2 +-
 .../spi/impl/OzoneManagerServiceProviderImpl.java  |  32 +-
 .../ozone/recon/tasks/ContainerKeyMapperTask.java  |   8 +-
 .../ozone/recon/tasks/FileSizeCountTask.java       |  12 +-
 .../ozone/recon/tasks/OMDBUpdatesHandler.java      |  79 ++-
 .../ozone/recon/tasks/ReconTaskController.java     |   3 +-
 .../ozone/recon/tasks/ReconTaskControllerImpl.java |  70 ++-
 .../ozone/recon/api/TestTaskStatusService.java     |  86 +++
 .../recon/tasks/TestReconTaskControllerImpl.java   |  38 +-
 hadoop-ozone/s3gateway/pom.xml                     |   5 +-
 .../hadoop/ozone/s3/endpoint/BucketEndpoint.java   |   4 +-
 .../hadoop/ozone/s3/endpoint/ObjectEndpoint.java   |   8 +-
 .../hadoop/ozone/s3/endpoint/RootEndpoint.java     |   4 +-
 .../hadoop/ozone/s3/exception/OS3Exception.java    |   2 +-
 .../s3/header/AuthenticationHeaderParser.java      |   4 +
 .../ozone/s3/header/AuthorizationHeaderV4.java     |  18 +-
 .../hadoop/ozone/s3/endpoint/TestBucketPut.java    | 114 ++++
 hadoop-ozone/tools/pom.xml                         |   1 -
 .../hadoop/ozone/freon/BaseFreonGenerator.java     | 116 ++--
 .../hadoop/ozone/freon/RandomKeyGenerator.java     |   8 +-
 hadoop-ozone/upgrade/pom.xml                       |   1 -
 pom.xml                                            |  71 +--
 365 files changed, 7053 insertions(+), 4472 deletions(-)
 copy hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/{utils/MetaStoreIterator.java
=> recon/ReconConfigKeys.java} (58%)
 copy {hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/scm/cli => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon}/package-info.java
(90%)
 create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ContainerCommandResponseBuilders.java
 create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/IncrementalChunkBuffer.java
 create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/protocol/MockDatanodeDetails.java
 copy hadoop-ozone/recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ReconSchemaDefinition.java
=> hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetUtils.java (56%)
 create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/pipeline/MockPipeline.java
 create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/MockGatheringChannel.java
 rename hadoop-hdds/common/src/test/resources/{log4j2.properties => auditlog.properties}
(100%)
 delete mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/SmallFileUtils.java
 copy hadoop-hdds/docs/{archetypes/default.md => content/_index.zh.md} (93%)
 copy hadoop-hdds/docs/{archetypes/default.md => content/concept/Datanodes.zh.md} (88%)
 copy hadoop-hdds/docs/{archetypes/default.md => content/concept/_index.zh.md} (92%)
 copy hadoop-hdds/docs/{static/NOTES.md => themes/ozonedoc/layouts/partials/languages.html}
(77%)
 copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartUploadList.java
=> hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/OzoneStorageContainerManager.java
(54%)
 create mode 100644 hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeWaitSubcommand.java
 copy hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationException.java
=> hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMLeaderNotReadyException.java
(65%)
 rename hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/{NotLeaderException.java
=> OMNotLeaderException.java} (71%)
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/.env
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config
 create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/README.md
 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-ockg.yaml (71%)
 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-rk.yaml (72%)
 rename hadoop-ozone/dist/src/main/compose/{ozones3/docker-config => ozone/monitoring.conf}
(76%)
 create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/monitoring.yaml
 rename hadoop-ozone/dist/src/main/compose/{ozone-recon/test.sh => ozone/profiling.yaml}
(68%)
 mode change 100755 => 100644
 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/prometheus.yml (72%)
 copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/startOM.sh => ozone/run.sh} (76%)
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/.env
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/README.md
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
 delete mode 100755 hadoop-ozone/dist/src/main/compose/ozoneperf/test.sh
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/.env
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/docker-compose.yaml
 delete mode 100755 hadoop-ozone/dist/src/main/compose/ozones3/test.sh
 copy {hadoop-hdds/tools => hadoop-ozone/fault-injection-test/mini-chaos-tests}/pom.xml
(68%)
 create mode 100755 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/bin/start-chaos.sh
 rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java
(97%)
 rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
(66%)
 rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
(90%)
 create mode 100644 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils/LoadBucket.java
 rename hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/chaos => fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils}/TestProbability.java
(97%)
 copy hadoop-ozone/{tools/src/main => fault-injection-test/mini-chaos-tests/src/test}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
(100%)
 copy hadoop-ozone/{ozonefs => fault-injection-test/mini-chaos-tests}/src/test/resources/log4j.properties
(75%)
 delete mode 100755 hadoop-ozone/integration-test/src/test/bin/start-chaos.sh
 create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
 create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneHACluster.java
 delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisClient.java
 delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandler.java
 delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandlerImpl.java
 create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMStorage.java
 create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestTrashService.java
 create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java
 create mode 100644 hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestReadWriteStatistics.java
 create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconSchemaManager.java
 copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{UtilizationService.java
=> TaskStatusService.java} (60%)
 create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconDatanodeProtocolServer.java
 copy hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/PendingDeleteHandler.java
=> hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconNodeManager.java
(54%)
 create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManager.java
 copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/{recovery => scm}/package-info.java
(95%)
 create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestTaskStatusService.java
 create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java


---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org


Mime
View raw message