hadoop-ozone-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From e...@apache.org
Subject [hadoop-ozone] branch HDDS-1880-Decom updated (920b8c5 -> 92e8d47)
Date Thu, 12 Dec 2019 16:30:40 GMT
This is an automated email from the ASF dual-hosted git repository.

elek pushed a change to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.


    from 920b8c5  HDDS-2459. Refactor ReplicationManager to consider maintenance states
     add 678e4bc  HDDS-2454. Improve OM HA robot tests. (#136)
     add 46a0875  HDDS-2510. Use isEmpty() to check whether the collection is empty or not
in Ozone Manager module. (#258)
     add 2185ac7  HDDS-2591. No tailMap needed for startIndex 0 in ContainerSet#listContainer
(#254)
     add 52579a2  HDDS-2595. Update Ratis version to latest snapshot version (#251)
     add 4a91745  HDDS-2609. Move plain unit tests out of integration tests (#260)
     add 15b05fa  HDDS-2626. Avoid hostname lookup for invalid local IP addresses (#267)
     add 728755e  HDDS-2627. Skip sonar check in forks (#266)
     add 3de23cc  Revert "HDDS-2595. Update Ratis version to latest snapshot version (#251)"
(#269)
     add a637a27  HDDS-2477. TableCache cleanup issue for OM non-HA. (#159)
     add 4f36984  HDDS-2602 Make auto creation for RATIS ONE pipeline configurable. (#248)
     add fe7fccf  HDDS-2372. Datanode pipeline is failing with NoSuchFileException (#166)
     add 045e3da  HDDS-2635 Minor:Fix typo in HddsConfigKeys and ozone-default. (#274)
     add ba9db10  HDDS-2632. Fix TestContainerPersistence#testDeleteChunk (#273)
     add ac93176  HDDS-2633. Let findbugs.sh skip frontend plugin for Recon (#275)
     add 0ef75b0  HDDS-2528. Sonar : code smell category issues in CommitWatcher (#247)
     add a73c470  HDDS-2034. Async RATIS pipeline creation and destroy through heartbeat commands
     add cd892e0  HDDS-2640 Add leaderID information in pipeline list subcommand (#279)
     add b1ceedd  Revert "HDDS-2640 Add leaderID information in pipeline list subcommand (#279)"
(#280)
     add c91916d  HDDS-2639. Turning off unit test until review of HDDS-2639
     add 72ab0a1  HDDS-2648. Disabling flaky test unil the fix.
     add cd3f8c9  HDDS-2640 Add leaderID information in pipeline list subcommand (#281)
     add 33b3c16  HDDS-2628. Make AuditMessage parameters strongly typed (#277)
     add 5e49d7a  HDDS-2505. Fix logic related to SCM address calculation in HddsUtils
     add 46fd24b  HDDS-2444. Remove server side dependencies from ozonefs jar files
     add 968dec4  HDDS-2619. Fix Sonar issues in ReconTaskControllerImpl
     add afda1c4  HDDS-2655. Use pre-compiled Pattern in NetUtils#normalize
     add 7524ee4  HDDS-2658. Insight log level reset does not work
     add 3480c89  HDDS-2654. Conditionally enable profiling at the kernel level
     add a25b2b4  HDDS-2639. TestTableCacheImpl is flaky (#278)
     add f141097  HDDS-2657. Key get command creates the output file even in case of KEY_NOT_FOUND
(#290)
     add 4d42714  HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader (#137)
     add 87e02a3  HDDS-2573. Handle InterruptedException in KeyOutputStream. (#257)
     add 9381dd0  HDDS-2637. Handle LeaderNot ready exception in OzoneManager StateMachine
and upgrade ratis to latest version. (#276)
     add 2fb8747  HDDS-2386. Implement incremental ChunkBuffer. (#205)
     add dbd05b0  HDDS-2620. Fix listMultipartupload API. (#263)
     add ea6c2c9  HDDS-1722. Use the bindings in ReconSchemaGenerationModule to create Recon
SQL tables on startup. (#294)
     add a3a2bfd  HDDS-2571. Handle InterruptedException in SCMPipelineManager (#295)
     add c9f2b02  HDDS-2656. Prefer execute() over submit() if the returned Future is ignored
(#302)
     add 7b71678  HDDS-2475. Unregister ContainerMetadataScrubberMetrics on thread exit (#286)
     add 5aa7383  HDDS-2653. Improve executor memory usage in new Freon tests (#284)
     add f686239  HDDS-2608. Provide command to wait until SCM is out from the safe-mode (#250)
     add 0ff53ef  HDDS-2646. Start acceptance tests only if at least one THREE pipeline is
available (#282)
     add a9990f6  HDDS-2622. Add config parameter for setting limit on total bytes of pending
requests in Ratis. (#264)
     add 8e1e225  HDDS-2667. Prometheus reports invalid metric type
     add 6c1a9ff  Addendum HDDS-2646. Start acceptance tests only if at least one THREE pipeline
is available (#282)
     add b35ad98  HDDS-2506. Remove keyAllocationInfo and replication info from the auditLog
(#288)
     add 4f1869c  HDDS-2565. Handle InterruptedException in VolumeSet
     add 1998983  HDDS-2600. Move chaos test to org.apache.hadoop.ozone.chaos package. (#307)
     add 2f72319  HDDS-1873. Add API to get last completed times for every Recon task. (#233)
     add e359e66  HDDS-2666. Suppress loader constraint violation message in TestOzoneFileSystemWithMocks
(#303)
     add 1af4f48  HDDS-2530. Sonar : refactor verifyResourceName in HddsClientUtils to fix
Sonar errors (#293)
     add d18aa99  HDDS-2629. Ozone CLI: CreationTime/modifyTime of volume/bucket/key in…
(#272)
     add 9b07c34  HDDS-2677. Acceptance test may fail despite success status (#312)
     add 1e18347  HDDS-2570. Handle InterruptedException in ProfileServlet. (#308)
     add 9593b25  HDDS-1987. Fix listStatus API (#108)
     add f69cb1e  HDDS-2636. Refresh pipeline information in OzoneManager lookupFile call.
(#301)
     add 76ad638  HDDS-2678. Add thread name to log pattern (#315)
     add e54b9be  HDDS-2662. Update gRPC and datanode protobuf version in Ozone
     add db4a6e6  HDDS-2681. Add leak detection memory flags to MiniOzoneChaosCluster
     add 5eef651  HDDS-1991. Remove RatisClient in OM HA. (#319)
     add 1503ab8  HDDS-2692. Seek to file end throws EOF Exception. (#326)
     add 588ff1e  HDDS-2689. OMException NOT_A_FILE missing space in the exception message.
(#327)
     add 91a6477  HDDS-2687. Fix sonar issues in package org.apache.hadoop.ozone.recon.api.
(#325)
     add f3abb4c  HDDS-2668. Sonar : fix issues reported in BlockManagerImpl (#317)
     add 1d56bc2  HDDS-2645. Refactor MiniOzoneChaosCluster to a different package to add
filesystem tests. (#292)
     add 5d9ae67  HDDS-2683. Ratis MaxBuffer should be the same size as the segment size.
(#320)
     add 8bfc726  HDDS-1993. Merge OzoneManagerRequestHandler and OzoneManagerHARequest…
(#304)
     add 022ceb5  HDDS-2693. HddsVolume mixes ChunkLayOutVersion and DataNodeLayoutVersion
(#330)
     add 42a30da  HDDS-2105. Merge OzoneClientFactory#getRpcClient functions (#148)
     add 6b78cc5  HDDS-2389. add toStateMachineLogEntryString provider in Ozone's ContainerStateMachine.
(#298)
     add 51c5e42  HDDS-2532. Sonar : fix issues in OzoneQuota (#311)
     add 050d58b  HDDS-2561. Handle InterruptedException in LeaseManager (#338)
     add 15f4ed4  HDDS-2704. Source tar file is not created during the relase build (#334)
     add 10bff97  HDDS-2711. OM does not report JVM metrics (#346)
     add 6506789  HDDS-2688. Refactor container response builders to hadoop-hdds/common (#324)
     add 0b1ea25  HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception
on container creation failure (#296).
     add e14f709  HDDS-2588. Consolidate compose environments
     new 92e8d47  Merge remote-tracking branch 'origin/master' into HDDS-1880-Decom

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/workflows/post-commit.yml                  |   1 +
 .../hadoop/hdds/scm/client/HddsClientUtils.java    |  75 +--
 .../hadoop/hdds/scm/storage/BlockOutputStream.java |   3 +-
 .../apache/hadoop/hdds/scm/storage/BufferPool.java |   4 +-
 .../hadoop/hdds/scm/storage/CommitWatcher.java     |  20 +-
 hadoop-hdds/common/pom.xml                         |   9 +-
 .../org/apache/hadoop/hdds/HddsConfigKeys.java     |  10 +-
 .../java/org/apache/hadoop/hdds/HddsUtils.java     | 216 ++++----
 .../org/apache/hadoop/hdds/client/OzoneQuota.java  |  12 +-
 .../org/apache/hadoop/hdds/scm/ScmConfigKeys.java  |  13 +-
 .../org/apache/hadoop/hdds/scm/net/NetUtils.java   |  19 +-
 .../apache/hadoop/hdds/scm/pipeline/Pipeline.java  |   8 +-
 .../ContainerCommandResponseBuilders.java          | 269 +++++++++
 .../org/apache/hadoop/hdds/utils/Scheduler.java    |   7 +-
 .../apache/hadoop/hdds/utils/db/DBCheckpoint.java  |  12 +
 .../hadoop/hdds/utils/db/RocksDBCheckpoint.java    |  11 +
 .../org/apache/hadoop/hdds/utils/db/Table.java     |  10 +-
 .../apache/hadoop/hdds/utils/db/TypedTable.java    |   5 +-
 .../hadoop/hdds/utils/db/cache/EpochEntry.java     |   2 +-
 .../hadoop/hdds/utils/db/cache/TableCache.java     |  14 +-
 .../hadoop/hdds/utils/db/cache/TableCacheImpl.java |  58 +-
 .../org/apache/hadoop/ozone/OzoneConfigKeys.java   |   5 +
 .../java/org/apache/hadoop/ozone/OzoneConsts.java  |   2 +
 .../org/apache/hadoop/ozone/OzoneSecurityUtil.java |  53 +-
 .../apache/hadoop/ozone/audit/AuditMessage.java    |  51 +-
 .../apache/hadoop/ozone/common/ChunkBuffer.java    |  49 +-
 .../common/ChunkBufferImplWithByteBuffer.java      |  10 +-
 .../ozone/common/IncrementalChunkBuffer.java       | 267 +++++++++
 .../ozone/container/common/helpers/BlockData.java  |   5 -
 .../apache/hadoop/ozone/lease/LeaseManager.java    |   4 +-
 .../apache/hadoop/ozone/web/utils/JsonUtils.java   |  15 +-
 .../common/src/main/resources/ozone-default.xml    |  69 +--
 .../apache/hadoop/hdds/scm/net/TestNetUtils.java   |  30 +-
 .../hdds/utils/db/TestTypedRDBTableStore.java      |   7 +-
 .../hdds/utils/db/cache/TestTableCacheImpl.java    | 244 ++++++++-
 .../hadoop/ozone/audit/TestOzoneAuditLogger.java   |  22 +-
 .../hadoop/ozone/common/TestChunkBuffer.java       |  18 +
 .../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 117 ++--
 .../container/common/helpers/ContainerUtils.java   |  85 +--
 .../ozone/container/common/impl/ContainerSet.java  |   2 +-
 .../container/common/impl/HddsDispatcher.java      |  18 +-
 .../common/statemachine/DatanodeStateMachine.java  |   6 +
 .../CloseContainerCommandHandler.java              |  11 +-
 .../ClosePipelineCommandHandler.java               | 120 ++++
 .../commandhandler/CommandHandler.java             |   2 +-
 .../CreatePipelineCommandHandler.java              | 135 +++++
 .../states/endpoint/HeartbeatEndpointTask.java     |  22 +
 .../common/transport/server/XceiverServerSpi.java  |  18 +
 .../server/ratis/ContainerStateMachine.java        |  30 +-
 .../transport/server/ratis/XceiverServerRatis.java |  78 ++-
 .../ozone/container/common/volume/HddsVolume.java  |   3 +-
 .../ozone/container/common/volume/VolumeSet.java   |   1 +
 .../ozone/container/keyvalue/KeyValueHandler.java  | 124 ++---
 .../container/keyvalue/helpers/BlockUtils.java     |  88 +--
 .../container/keyvalue/helpers/ChunkUtils.java     |  62 +--
 .../keyvalue/helpers/KeyValueContainerUtil.java    |  27 -
 .../container/keyvalue/helpers/SmallFileUtils.java |  91 ---
 .../container/keyvalue/impl/BlockManagerImpl.java  |  65 +--
 .../container/keyvalue/impl/ChunkManagerImpl.java  |  43 +-
 .../container/ozoneimpl/ContainerController.java   |   9 +
 .../ozoneimpl/ContainerDataScrubberMetrics.java    |   8 +-
 .../ozoneimpl/ContainerMetadataScanner.java        |  18 +-
 .../ContainerMetadataScrubberMetrics.java          |   8 +-
 .../replication/ReplicationSupervisor.java         |   2 +-
 ...ainerCommand.java => ClosePipelineCommand.java} |  53 +-
 .../protocol/commands/CreatePipelineCommand.java   | 100 ++++
 .../proto/StorageContainerDatanodeProtocol.proto   |  23 +
 .../ozone/container/ContainerTestHelper.java       | 325 +----------
 .../container/common/helpers/TestBlockData.java    |   0
 .../impl/TestContainerDeletionChoosingPolicy.java  |   0
 .../common/impl/TestContainerPersistence.java      |   2 +-
 .../container/common/impl/TestContainerSet.java    |  39 +-
 .../container/common/impl/TestHddsDispatcher.java  |   6 +-
 .../container/common/volume/TestVolumeSet.java     |  24 +
 .../container/keyvalue/TestChunkManagerImpl.java   |   3 +-
 .../container/keyvalue/helpers/TestChunkUtils.java |  39 +-
 .../ozoneimpl/TestContainerScrubberMetrics.java    |  35 +-
 .../apache/hadoop/hdds/server/BaseHttpServer.java  |  19 +-
 .../apache/hadoop/hdds/server/ProfileServlet.java  |   7 +-
 .../hadoop/hdds/server/PrometheusMetricsSink.java  |   5 +-
 .../org/apache/hadoop/hdds/server/ServerUtils.java |  15 +
 .../hdds/server/TestPrometheusMetricsSink.java     |   2 +-
 .../hadoop/hdds/scm/block/BlockManagerImpl.java    |   3 +
 .../hdds/scm/container/ContainerStateManager.java  |   1 +
 .../apache/hadoop/hdds/scm/events/SCMEvents.java   |  12 +-
 .../scm/pipeline/BackgroundPipelineCreator.java    |  23 +-
 .../hadoop/hdds/scm/pipeline/PipelineFactory.java  |  13 +-
 .../hadoop/hdds/scm/pipeline/PipelineManager.java  |  15 +-
 .../hadoop/hdds/scm/pipeline/PipelineProvider.java |   2 +
 .../hdds/scm/pipeline/PipelineReportHandler.java   |  54 +-
 .../hdds/scm/pipeline/PipelineStateManager.java    |   4 +-
 .../hdds/scm/pipeline/RatisPipelineProvider.java   | 137 ++---
 .../hdds/scm/pipeline/RatisPipelineUtils.java      | 103 ----
 .../hdds/scm/pipeline/SCMPipelineManager.java      |  83 ++-
 .../hdds/scm/pipeline/SCMPipelineMetrics.java      |  10 +
 .../hdds/scm/pipeline/SimplePipelineProvider.java  |   5 +
 .../scm/safemode/HealthyPipelineSafeModeRule.java  |  80 +--
 .../safemode/OneReplicaPipelineSafeModeRule.java   |  65 +--
 .../hdds/scm/safemode/SCMSafeModeManager.java      |  16 +-
 .../hadoop/hdds/scm/safemode/SafeModeHandler.java  |   5 +-
 .../hdds/scm/server/SCMBlockProtocolServer.java    |  26 +-
 .../hdds/scm/server/SCMClientProtocolServer.java   |  30 +-
 .../scm/server/SCMDatanodeHeartbeatDispatcher.java |   1 +
 .../hdds/scm/server/SCMDatanodeProtocolServer.java |  46 +-
 .../hdds/scm/server/StorageContainerManager.java   |  10 +-
 .../java/org/apache/hadoop/hdds/scm/TestUtils.java |  12 +
 .../hadoop/hdds/scm/block/TestBlockManager.java    |  44 +-
 .../container/TestCloseContainerEventHandler.java  |  11 +-
 .../scm/container/TestSCMContainerManager.java     |   4 +-
 .../hdds/scm/node/TestContainerPlacement.java      |   2 +-
 .../hadoop/hdds/scm/node/TestDeadNodeHandler.java  |  11 +-
 .../hadoop/hdds/scm/node/TestSCMNodeManager.java   |   9 +-
 .../scm/pipeline/MockRatisPipelineProvider.java    |  10 +-
 .../scm/pipeline/TestPipelineStateManager.java     |   0
 .../scm/pipeline/TestRatisPipelineProvider.java    |  26 +-
 .../hdds/scm/pipeline/TestSCMPipelineManager.java  |  93 +++-
 .../scm/pipeline/TestSimplePipelineProvider.java   |   0
 .../hdds/scm/pipeline}/pipeline/package-info.java  |   0
 .../safemode/TestHealthyPipelineSafeModeRule.java  |  44 +-
 .../TestOneReplicaPipelineSafeModeRule.java        |  36 +-
 .../hdds/scm/safemode/TestSCMSafeModeManager.java  |  50 +-
 .../hadoop/hdds/scm/cli/SafeModeCommands.java      |   1 +
 .../hdds/scm/cli/SafeModeWaitSubcommand.java       |  93 ++++
 .../apache/hadoop/ozone/client/OzoneBucket.java    |  11 +-
 .../hadoop/ozone/client/OzoneClientFactory.java    |  68 +--
 .../org/apache/hadoop/ozone/client/OzoneKey.java   |  14 +-
 .../apache/hadoop/ozone/client/OzoneVolume.java    |   9 +-
 .../hadoop/ozone/client/io/KeyInputStream.java     |   2 +-
 .../hadoop/ozone/client/io/KeyOutputStream.java    |   1 +
 .../hadoop/ozone/client/TestHddsClientUtils.java   |  46 ++
 hadoop-ozone/common/pom.xml                        |   8 -
 .../main/java/org/apache/hadoop/ozone/OmUtils.java |  61 +--
 .../org/apache/hadoop/ozone/om/OMConfigKeys.java   |   9 -
 .../apache/hadoop/ozone/om/OMMetadataManager.java  |   3 +-
 .../om/exceptions/OMLeaderNotReadyException.java   |  22 +-
 ...derException.java => OMNotLeaderException.java} |  32 +-
 .../ozone/om/ha/OMFailoverProxyProvider.java       |   2 +-
 .../hadoop/ozone/om/helpers/OMRatisHelper.java     |  43 --
 .../apache/hadoop/ozone/om/helpers/OmKeyArgs.java  |   2 -
 .../ozone/om/protocol/OzoneManagerHAProtocol.java  |   6 +-
 ...OzoneManagerProtocolClientSideTranslatorPB.java |  51 +-
 .../java/org/apache/hadoop/ozone/TestOmUtils.java  |  65 +--
 hadoop-ozone/csi/pom.xml                           |  11 +-
 hadoop-ozone/dev-support/checks/findbugs.sh        |   8 +-
 .../dist/dev-support/bin/dist-tar-stitching        |   6 +-
 hadoop-ozone/dist/pom.xml                          |  39 ++
 .../dist/src/main/compose/ozone-mr/common-config   |   1 +
 .../compose/ozone-mr/hadoop27/docker-compose.yaml  |   1 +
 .../compose/ozone-mr/hadoop31/docker-compose.yaml  |   1 +
 .../compose/ozone-mr/hadoop32/docker-compose.yaml  |   1 +
 .../compose/ozone-om-ha-s3/docker-compose.yaml     |   1 +
 .../dist/src/main/compose/ozone-recon/.env         |  18 -
 .../main/compose/ozone-recon/docker-compose.yaml   |  65 ---
 .../src/main/compose/ozone-recon/docker-config     |  35 --
 .../compose/ozone-topology/docker-compose.yaml     |   1 +
 .../src/main/compose/ozone-topology/docker-config  |   1 +
 hadoop-ozone/dist/src/main/compose/ozone/README.md |  99 ++++
 .../src/main/compose/ozone/docker-compose.yaml     |  91 +--
 .../dist/src/main/compose/ozone/docker-config      |  11 +-
 .../compose/{ozoneperf => ozone}/freon-ockg.yaml   |  19 +-
 .../compose/{ozoneperf => ozone}/freon-rk.yaml     |  19 +-
 .../docker-config => ozone/monitoring.conf}        |  15 +-
 .../dist/src/main/compose/ozone/monitoring.yaml    |  56 ++
 .../{ozone-recon/test.sh => ozone/profiling.yaml}  |  30 +-
 .../compose/{ozoneperf => ozone}/prometheus.yml    |  20 +-
 .../{ozone-om-ha/startOM.sh => ozone/run.sh}       |  13 +-
 hadoop-ozone/dist/src/main/compose/ozone/test.sh   |   9 +-
 .../src/main/compose/ozoneblockade/docker-config   |   2 +
 hadoop-ozone/dist/src/main/compose/ozoneperf/.env  |  18 -
 .../dist/src/main/compose/ozoneperf/README.md      |  74 ---
 .../src/main/compose/ozoneperf/docker-compose.yaml |  84 ---
 .../dist/src/main/compose/ozoneperf/docker-config  |  36 --
 .../dist/src/main/compose/ozoneperf/test.sh        |  30 -
 .../compose/ozones3-haproxy/docker-compose.yaml    |   1 +
 hadoop-ozone/dist/src/main/compose/ozones3/.env    |  18 -
 .../src/main/compose/ozones3/docker-compose.yaml   |  58 --
 hadoop-ozone/dist/src/main/compose/ozones3/test.sh |  34 --
 .../src/main/compose/ozonescripts/docker-config    |   3 +-
 .../compose/ozonesecure-mr/docker-compose.yaml     |   1 +
 .../main/compose/ozonesecure/docker-compose.yaml   |   1 +
 hadoop-ozone/dist/src/main/compose/testlib.sh      |  52 +-
 .../dist/src/main/dockerlibexec/entrypoint.sh      |   8 +
 .../dist/src/main/smoketest/__init__.robot         |   3 +-
 .../dist/src/main/smoketest/basic/basic.robot      |   4 +-
 .../dist/src/main/smoketest/omha/testOMHA.robot    |  28 +-
 hadoop-ozone/dist/src/shell/conf/log4j.properties  |  10 +-
 .../fault-injection-test/mini-chaos-tests}/pom.xml |  37 +-
 .../mini-chaos-tests/src/test/bin/start-chaos.sh   |  46 ++
 .../apache/hadoop/ozone/MiniOzoneChaosCluster.java |   6 +-
 .../hadoop/ozone/MiniOzoneLoadGenerator.java       | 107 +---
 .../hadoop/ozone/TestMiniChaosOzoneCluster.java    |  12 +-
 .../org/apache/hadoop/ozone/utils/LoadBucket.java  |  99 ++++
 .../hadoop/ozone/utils}/TestProbability.java       |   2 +-
 .../services/org.apache.hadoop.fs.FileSystem       |   0
 .../src/test/resources/log4j.properties            |   8 +-
 hadoop-ozone/fault-injection-test/pom.xml          |   1 +
 .../apache/hadoop/ozone/insight/LogSubcommand.java |  31 +-
 .../integration-test/src/test/bin/start-chaos.sh   |  35 --
 .../TestContainerStateManagerIntegration.java      |   6 +-
 .../metrics/TestSCMContainerManagerMetrics.java    |   3 +
 .../hdds/scm/pipeline/TestPipelineClose.java       |   4 +-
 .../TestRatisPipelineCreateAndDestroy.java         |  29 +
 .../scm/pipeline/TestRatisPipelineProvider.java    | 210 +++++++
 .../org/apache/hadoop/ozone/MiniOzoneCluster.java  |   2 +-
 .../apache/hadoop/ozone/MiniOzoneClusterImpl.java  |  53 +-
 .../hadoop/ozone/MiniOzoneHAClusterImpl.java       |  24 +-
 .../hadoop/ozone/TestContainerOperations.java      |   2 +-
 .../TestContainerStateMachineIdempotency.java      |   2 +-
 .../apache/hadoop/ozone/TestMiniOzoneCluster.java  |   6 +-
 .../hadoop/ozone/TestMiniOzoneHACluster.java       | 112 ++++
 .../hadoop/ozone/TestStorageContainerManager.java  |  18 +-
 .../apache/hadoop/ozone/client/rpc/TestBCSID.java  |   3 +
 .../ozone/client/rpc/TestBlockOutputStream.java    |   5 +-
 .../rpc/TestBlockOutputStreamWithFailures.java     |  13 +-
 .../rpc/TestCloseContainerHandlingByClient.java    |   7 +-
 .../hadoop/ozone/client/rpc/TestCommitWatcher.java |   4 +-
 .../rpc/TestContainerReplicationEndToEnd.java      |   4 +-
 .../client/rpc/TestContainerStateMachine.java      |  10 +-
 .../rpc/TestContainerStateMachineFailures.java     |  18 +-
 .../client/rpc/TestFailureHandlingByClient.java    |   7 +-
 .../ozone/client/rpc/TestKeyInputStream.java       |   7 +-
 .../rpc/TestMultiBlockWritesWithDnFailures.java    |   5 +-
 .../client/rpc/TestOzoneAtRestEncryption.java      |  14 +-
 .../rpc/TestOzoneClientRetriesOnException.java     |   9 +-
 .../client/rpc/TestOzoneRpcClientAbstract.java     |  90 ++-
 .../ozone/client/rpc/TestSecureOzoneRpcClient.java |   8 +-
 .../ozone/client/rpc/TestWatchForCommit.java       |   7 +-
 .../ozone/container/TestContainerReplication.java  |   2 +-
 .../apache/hadoop/ozone/container/TestHelper.java  | 305 +++++++++++
 .../commandhandler/TestBlockDeletion.java          |   6 +-
 .../commandhandler/TestCloseContainerHandler.java  |   2 +
 .../commandhandler/TestDeleteContainerHandler.java |   2 +
 .../hadoop/ozone/dn/scrubber/TestDataScrubber.java |  10 +-
 .../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 235 ++++++++
 .../org/apache/hadoop/ozone/om/TestKeyPurging.java |   5 +-
 .../hadoop/ozone/om/TestOMRatisSnapshots.java      |  20 +-
 .../apache/hadoop/ozone/om/TestOzoneManagerHA.java |  34 +-
 .../apache/hadoop/ozone/om/TestScmSafeMode.java    |   2 +-
 .../hadoop/ozone/ozShell/TestOzoneShellHA.java     |  20 +-
 .../hadoop/ozone/scm/TestContainerSmallFile.java   |   2 +-
 .../scm/TestGetCommittedBlockLengthAndPutKey.java  |   2 +-
 .../org/apache/hadoop/ozone/scm/TestSCMMXBean.java |   2 +-
 .../hadoop/ozone/scm/node/TestSCMNodeMetrics.java  |   4 +-
 hadoop-ozone/ozone-manager/pom.xml                 |   5 +
 .../apache/hadoop/ozone/om/KeyDeletingService.java |   2 +-
 .../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 176 ++++--
 .../hadoop/ozone/om/OMDBCheckpointServlet.java     |  19 +-
 .../java/org/apache/hadoop/ozone/om/OMStorage.java |  17 +-
 .../hadoop/ozone/om/OmMetadataManagerImpl.java     |  35 +-
 .../org/apache/hadoop/ozone/om/OzoneManager.java   | 103 +---
 .../hadoop/ozone/om/ratis/OMRatisSnapshotInfo.java |  15 +-
 .../ozone/om/ratis/OzoneManagerDoubleBuffer.java   |  16 +-
 .../ozone/om/ratis/OzoneManagerRatisClient.java    | 210 -------
 .../ozone/om/ratis/OzoneManagerRatisServer.java    |  83 ++-
 .../ozone/om/ratis/OzoneManagerStateMachine.java   |  85 ++-
 .../hadoop/ozone/om/request/OMClientRequest.java   |   6 +-
 .../hadoop/ozone/om/request/RequestAuditor.java    |   3 -
 .../ozone/om/request/file/OMFileCreateRequest.java |   6 +-
 .../om/snapshot/OzoneManagerSnapshotProvider.java  |  10 +
 .../protocolPB/OzoneManagerHARequestHandler.java   |  39 --
 .../OzoneManagerHARequestHandlerImpl.java          |  56 --
 ...OzoneManagerProtocolServerSideTranslatorPB.java |  27 +-
 .../protocolPB/OzoneManagerRequestHandler.java     | 607 +--------------------
 .../hadoop/ozone/protocolPB/RequestHandler.java    |  28 +-
 .../hadoop/ozone/web/ozShell/OzoneAddress.java     |   3 +-
 .../ozone/web/ozShell/keys/GetKeyHandler.java      |   4 +-
 .../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 256 ++++++++-
 .../org/apache/hadoop/ozone/om/TestOMStorage.java  |  91 +++
 .../ozone/om/ratis}/TestOMRatisSnapshotInfo.java   |   8 +-
 ...TestOzoneManagerDoubleBufferWithOMResponse.java |   1 +
 .../om/ratis/TestOzoneManagerRatisServer.java      |  29 +-
 .../ozone/om/request/TestOMRequestUtils.java       |  17 +
 .../security/acl/TestOzoneNativeAuthorizer.java    |  20 +-
 .../ozone/web/ozShell/TestObjectPrinter.java       |   6 +-
 hadoop-ozone/ozonefs-lib-current/pom.xml           |  15 -
 .../fs/ozone/BasicOzoneClientAdapterImpl.java      |   2 +-
 .../hadoop/fs/ozone/TestOzoneFileSystem.java       |  12 +
 .../fs/ozone/TestOzoneFileSystemWithMocks.java     |   2 +
 .../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java  |  48 +-
 .../hadoop/fs/ozone/TestOzoneFsRenameDir.java      |   2 +-
 .../hadoop/ozone/recon/ReconControllerModule.java  |   4 -
 .../hadoop/ozone/recon/ReconSchemaManager.java     |  55 ++
 .../org/apache/hadoop/ozone/recon/ReconServer.java |  26 +-
 .../ozone/recon/api/ContainerKeyService.java       | 112 ++--
 ...lizationService.java => TaskStatusService.java} |  39 +-
 .../hadoop/ozone/recon/api/UtilizationService.java |   5 -
 .../recon/recovery/ReconOMMetadataManager.java     |   6 +
 .../recon/recovery/ReconOmMetadataManagerImpl.java |  11 +
 .../spi/impl/OzoneManagerServiceProviderImpl.java  |  32 +-
 .../ozone/recon/tasks/ReconTaskController.java     |   3 +-
 .../ozone/recon/tasks/ReconTaskControllerImpl.java |  70 ++-
 .../ozone/recon/api/TestTaskStatusService.java     |  86 +++
 .../recon/tasks/TestReconTaskControllerImpl.java   |  38 +-
 hadoop-ozone/s3gateway/pom.xml                     |   4 +
 .../hadoop/ozone/s3/endpoint/BucketEndpoint.java   |   4 +-
 .../hadoop/ozone/s3/endpoint/ObjectEndpoint.java   |   8 +-
 .../hadoop/ozone/s3/endpoint/RootEndpoint.java     |   4 +-
 .../hadoop/ozone/freon/BaseFreonGenerator.java     | 116 ++--
 .../hadoop/ozone/freon/RandomKeyGenerator.java     |   2 +-
 .../freon/TestFreonWithDatanodeFastRestart.java    |   4 +-
 .../hadoop/ozone/fsck/TestContainerMapper.java     |   2 +-
 pom.xml                                            |  64 +--
 302 files changed, 6094 insertions(+), 4584 deletions(-)
 create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ContainerCommandResponseBuilders.java
 create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/IncrementalChunkBuffer.java
 copy hadoop-ozone/recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ReconSchemaDefinition.java
=> hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetUtils.java (56%)
 create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/ClosePipelineCommandHandler.java
 create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CreatePipelineCommandHandler.java
 delete mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/SmallFileUtils.java
 copy hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/{CloseContainerCommand.java
=> ClosePipelineCommand.java} (53%)
 create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/CreatePipelineCommand.java
 rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java
(65%)
 rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/helpers/TestBlockData.java
(100%)
 rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDeletionChoosingPolicy.java
(100%)
 rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java
(99%)
 delete mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
 rename {hadoop-ozone/integration-test => hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelineStateManager.java
(100%)
 rename {hadoop-ozone/integration-test => hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
(78%)
 rename {hadoop-ozone/integration-test => hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSimplePipelineProvider.java
(100%)
 copy {hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm => hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline}/pipeline/package-info.java
(100%)
 create mode 100644 hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeWaitSubcommand.java
 copy hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationException.java
=> hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMLeaderNotReadyException.java
(65%)
 rename hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/{NotLeaderException.java
=> OMNotLeaderException.java} (71%)
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/.env
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config
 create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/README.md
 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-ockg.yaml (71%)
 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-rk.yaml (72%)
 rename hadoop-ozone/dist/src/main/compose/{ozones3/docker-config => ozone/monitoring.conf}
(76%)
 create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/monitoring.yaml
 rename hadoop-ozone/dist/src/main/compose/{ozone-recon/test.sh => ozone/profiling.yaml}
(68%)
 mode change 100755 => 100644
 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/prometheus.yml (72%)
 copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/startOM.sh => ozone/run.sh} (76%)
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/.env
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/README.md
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
 delete mode 100755 hadoop-ozone/dist/src/main/compose/ozoneperf/test.sh
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/.env
 delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/docker-compose.yaml
 delete mode 100755 hadoop-ozone/dist/src/main/compose/ozones3/test.sh
 copy {hadoop-hdds/tools => hadoop-ozone/fault-injection-test/mini-chaos-tests}/pom.xml
(68%)
 create mode 100755 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/bin/start-chaos.sh
 rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java
(97%)
 rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
(66%)
 rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
(90%)
 create mode 100644 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils/LoadBucket.java
 rename hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/chaos => fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils}/TestProbability.java
(97%)
 copy hadoop-ozone/{tools/src/main => fault-injection-test/mini-chaos-tests/src/test}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
(100%)
 copy hadoop-ozone/{ozonefs => fault-injection-test/mini-chaos-tests}/src/test/resources/log4j.properties
(75%)
 delete mode 100755 hadoop-ozone/integration-test/src/test/bin/start-chaos.sh
 create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java
 create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneHACluster.java
 create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestHelper.java
 delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisClient.java
 delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandler.java
 delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandlerImpl.java
 create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMStorage.java
 rename hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot =>
ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis}/TestOMRatisSnapshotInfo.java
(90%)
 rename hadoop-ozone/{integration-test => ozone-manager}/src/test/java/org/apache/hadoop/ozone/security/acl/TestOzoneNativeAuthorizer.java
(96%)
 create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconSchemaManager.java
 copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{UtilizationService.java
=> TaskStatusService.java} (60%)
 create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestTaskStatusService.java


---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org


Mime
View raw message