From ozone-commits-return-694-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Mon Feb 10 02:23:21 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 151F118065D for ; Mon, 10 Feb 2020 03:23:20 +0100 (CET) Received: (qmail 15702 invoked by uid 500); 10 Feb 2020 02:23:20 -0000 Mailing-List: contact ozone-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: ozone-commits@hadoop.apache.org Delivered-To: mailing list ozone-commits@hadoop.apache.org Received: (qmail 15685 invoked by uid 99); 10 Feb 2020 02:23:20 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Feb 2020 02:23:20 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id 89A7B8B694; Mon, 10 Feb 2020 02:23:19 +0000 (UTC) Date: Mon, 10 Feb 2020 02:23:18 +0000 To: "ozone-commits@hadoop.apache.org" Subject: [hadoop-ozone] branch HDDS-1564 updated (43845da -> d5666c5) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <158130139893.14422.10439083671831457371@gitbox.apache.org> From: sammichen@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: hadoop-ozone X-Git-Refname: refs/heads/HDDS-1564 X-Git-Reftype: branch X-Git-Oldrev: 43845daa30822c9fb259acaafc1748d633e76fc6 X-Git-Newrev: d5666c5e3dde1c3a4e8ceb888ffea2607a1ab254 X-Git-NotificationType: ref_changed X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. sammichen pushed a change to branch HDDS-1564 in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git. discard 43845da HDDS-2923 Add fall-back protection for rack awareness in pipeline creation. (#516) omit 09aac8e HDDS-2924. Fix Pipeline#nodeIdsHash collision issue. (#478) omit 3123cfa HDDS-2913 Update config names and CLI for multi-raft feature. (#462) omit 2ebf21b HDDS-2772 Better management for pipeline creation limitation. (#410) omit 129d464 HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI (#375) omit 5a653fd HDDS-2756. Handle pipeline creation failure in different way when it exceeds pipeline limit omit cc0060c Resolve rebase conflict. omit acc9640 HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test (#291) omit fced9c1 HDDS-2650 Fix createPipeline CLI and make it message based. (#370) omit 00f0f81 Revert "HDDS-2650 Fix createPipeline CLI. (#340)" omit 3796154 HDDS-2035 Implement datanode level CLI to reveal pipeline relation. (#348) omit a2c65a5 HDDS-2650 Fix createPipeline CLI. (#340) omit 667a0c0 Rebase Fix omit a6c1f2a HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline. (#237) omit f8c5eb0 HDDS-1569 Support creating multiple pipelines with same datanode. Contributed by Li Cheng. omit f9fe003 HDDS-2089: Add createPipeline CLI. (#1418) omit 7fe5881 HDDS-1571. Create an interface for pipeline placement policy to support network topologies. (#1395) omit f1d994e HDDS-1577. Add default pipeline placement policy implementation. (#1366) add 0a3f0f8 HDDS-2725. start/_index.md translated to Chinese (#397) add 8748498 HDDS-2590 Integration tests for Recon with Ozone Manager (#356) add 34683e4 HDDS-2834. Directly read into ByteBuffer if it has array (#413) add 245d335 HDDS-2853. NPE in OzoneContainer Start. (#419) add 20834c5 HDDS-2827. Add initial UI of Datanodes in Recon add 53ef7dc HDDS-2710. Add Filesystem functionality in MiniOzoneChaosCluster. (#344) add acc3dde HDDS-2854. Add a config for the write chunk executor queue limit. (#420) add 406428c HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro… (#331) add 4b256aa HDDS-2542. Race condition between read and write stateMachineData. (#310) add 2fa37ef HDDS-2779. Fix list volume for --start parameter. (#385) add 046a06f HDDS-2579. Ozone client should refresh pipeline info if reads from all Datanodes fail (#372) add 44b48b7 HDDS-2706. Add class CRLCodec - used for certificate revocation list. (#337) add 7e6b821 HDDS-2857. Ozone Recon fails to start while ozone install (#424) add 088d76e HDDS-2763. Handle Datanode registration in Recon. (#405) add eceef90 HDDS-2851. Some OM Ratis config properties missing from ozone-default.xml. (#421) add 25b4baf HDDS-2863. BindException in TestSCMRestart (#425) add dedbbab HDDS-2872. ozone.recon.scm.db.dirs missing from ozone-default.xml. (#434) add 92234d4 HDDS-2781. Add ObjectID and updateID to BucketInfo to avoid replaying transactions (#381) add ba8212c HDDS-2279. S3 commands not working on HA cluster. (#333) add 1abfe05 HDDS-2727. start/OnPrem.md translated to Chinese (#429) add becceb6 HDDS-2750. OzoneFSInputStream to support StreamCapabilities (#379) add 1c75951 HDDS-2828. Add initial UI of Pipelines in Recon add c17c497 HDDS-2861. Support Freon progressbar in non-interactive environment (#423) add a479743 HDDS-2868. Add ObjectID and UpdateID to OMKeyInfo. (#428) add 37730f0 HDDS-1812. Du while calculating used disk space reports that chunk files are file not found (#271) add b78d9b2 HDDS-2757. Annoying 'Failed to close the container' message because pipeline is removed from PipelineStateMap before close container command returned (#443) add 883a662 HDDS-2875. Add a config in ozone to tune max outstanding requests in … (#436) add 2b80e99 HDDS-2740. start/FromSource.md translation (#444) add 1423491 HDDS-2726. start/StartFromDockerHub.md translated to Chinese (#417) add 001bd1d HDDS-2859. Hugo error should be propagated to build (#446) add 1caf1e3 HDDS-2866. Intermittent failure in TestOzoneManagerRocksDBLogging (#427) add 71c24a8 HDDS-2897. Ozone recon Start failed due to Kerberos principal not being found. (#453) add 9bc433a HDDS-2904. Remove default dependencies from hadoop-hdds/pom.xml (#459) add 79aff8b HDDS-2905. Remove unusued BlockLocation protocol related classes (#460) add 5d27f45 HDDS-2846. Handle Datanode Pipeline & Container Reports reports in Recon. (#431) add 2ea9afa HDDS-2900. Avoid logging NPE when space usage check is not configured add 968ec62 HDDS-2910. OzoneManager startup failure with throwing unhelpful exception message add c29936a HDDS-2898. build failure due to hadoop-hdds-client test (#463) add e7afa41 HDDS-2896. Use regex to match with ratis properties when creating ratis client. (#457) add 2e92658 Revert "HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro… (#331)" (#470) add 73cb410 HDDS-2733. start/Kubernetes.md translated to Chinese (#439) add c74429d HDDS-2903. Use regex to match with ratis properties when creating ratis server. (#458) add 1e442f6 HDDS-2738. start/Minikube.md translated to Chinese (#440) add 01540d7 HDDS-2734. start/RunningViaDocker.md translated to Chinese (#442) add dc60d18 HDDS-2915. Unit check passes despite Maven error (#465) add f080c22 HDDS-2741. shell/_index.md translation (#473) add 935006b HDDS-2742. shell/Format.md translation (#474) add 803fa29 HDDS-2918. Increase timeout of safe-mode exit in acceptance tests add 0510b55 HDDS-2902. execute_robot_test on unknown/unavailable container should fail acceptance test add f98c9ed HDDS-2889. Make DBStore and RDBStore more commons add 5950224 HDDS-2901. List Trash - Fix Cluster Max Keys Check add 1f05261 HDDS-2849. OM and SCM Web-server report HTTP 404 error when accessing '/' after cluster runs for several weeks add ab557db HDDS-2920. Remove ozone ratis client specific config keys. (#472) add 09df00f Revert "HDDS-2920. Remove ozone ratis client specific config keys. (#472)" (#482) add 4d603ec HDDS-2864. TestOMDbCheckpointServlet fails due to real Recon add c8bc3b7 HDDS-2895. Generate only the required keytabs for docker based secure tests (#455) add acd4619 HDDS-2743. shell/VolumeCommands.md translation (#475) add 15fb459 HDDS-2906. Added Unit Test Cases ofr CRLCodec (#471) add 3316aca HDDS-2937. Document bucket encryption option in shell/BucketCommands.md. (#483) add c013de2 HDDS-2938. Use regex to match with ratis grpc properties when creating ratis server. (#486) add d497bf6 HDDS-2920. Remove ozone ratis client specific config keys. (#484) add ae7c268 HDDS_2746. inteface/_index.md translation (#488) add 1372cc9 HDDS-2744. shell/BucketCommands.md translation (#476) add 25d1489 HDDS-2745. shell/KeyCommands.md translation (#477) add 4b8e8f6 HDDS-2888. Refactor Datanode StateContext to send reports and actions to all configured SCMs. (#447) add 4ce2796 HDDS-2921. Remove ozone ratis server specific config keys. (#485) add 242f503 HDDS-2925. Remove hdfs-client dependency from hdds-common add 345087f HDDS-2855. Recon getContainers API should return a maximum of 1000 containers by default. add ef79f33 HDDS-1335. Add basic UI for showing missing containers and keys add 66abf65 HDDS-2870. Handle replay of KeyCreate requests. (#448) add eb914ff HDDS-2916. OM HA cli getserviceroles not working. (#490) add cec238e HDDS-2926. Intermittent failure in TestRecon due to thread timing. (#481) add ee7b454 HDDS-2869. Handle pipeline bootstrap from SCM and create pipeline use case in Recon. (#466) add 6794e02 HDDS-2894. Handle replay of KeyDelete and KeyRename Requests (#452) add b53e3a5 HDDS-2931. Recon integration test should use ephemeral port for HTTP Server. (#500) add e234cd3 HDDS-2555. Handle InterruptedException in XceiverClientGrpc (#314) add a967de2 HDDS-2833. Enable integrations tests for github actions add 15cf3c8 HDDS-2956. Handle Replay of AllocateBlock request (#505) add bb57442 HDDS-2893. Handle replay of KeyPurge Request. (#450) add 7a3dcf9 HDDS-2850. Handle Create container use case in Recon. (#503) add 76523f2 HDDS-2747. interface/JavaApi.md (#489) add 0692413 HDDS-2748. interface/OzoneFS.md translation (#491) add 475b94a HDDS-2749. interface/S3.md translation (#493) add 61084e6 HDDS-2758. gdpr/_index.md translation (#494) add 5fb3219 HDDS-2952. Ensure ozone manager service user is part of ozone.adminis… (#506) add 0aa4b04 HDDS-2789. concept/_index.md translation (#496) add af43a28 HDDS-2883. Change the default client settings accordingly with change in default chunk size. (#487) add d1b8c08 HDDS-2973. Fix root deletion logic in delete API. (#517) add 486e872 HDDS-2892 Fixed typo in createClientRequest of OzoneManagerRatisUtils (#532) add b895670 HDDS-2977. ozonesecure acceptance test fails due to unexpected error message (#527) add 2d6d293 HDDS-2983. Acceptance test failures due to lack of disk space (#530) add cd361bf HDDS-2041. Don't depend on DFSUtil to check HTTP policy. (#533) add 8917d9a HDDS-2759. gdpr/GDPR in Ozone.md translation (#495) add 1ac8263 HDDS-2950. Upgrade jetty to the latest 9.4 release (#508) add 17bc776 HDDS-2845. Fix listing keys for setting --start with last key (#437) new 6ab3326 HDDS-1577. Add default pipeline placement policy implementation. (#1366) new 5cbfb37 HDDS-1571. Create an interface for pipeline placement policy to support network topologies. (#1395) new 3d6e5b1 HDDS-2089: Add createPipeline CLI. (#1418) new 1ef5e6d HDDS-1569 Support creating multiple pipelines with same datanode. Contributed by Li Cheng. new 4cbf554 HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline. (#237) new e98c12f Rebase Fix new e48ec92 HDDS-2650 Fix createPipeline CLI. (#340) new 81ed727 HDDS-2035 Implement datanode level CLI to reveal pipeline relation. (#348) new b5ef117 Revert "HDDS-2650 Fix createPipeline CLI. (#340)" new 6cdfdcb HDDS-2650 Fix createPipeline CLI and make it message based. (#370) new d6b9ec0 HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test (#291) new 4d463d6 Resolve rebase conflict. new facd403 HDDS-2756. Handle pipeline creation failure in different way when it exceeds pipeline limit new cb5486b HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI (#375) new b63a745 HDDS-2772 Better management for pipeline creation limitation. (#410) new 8b42cdc HDDS-2913 Update config names and CLI for multi-raft feature. (#462) new dd2b7e8 HDDS-2924. Fix Pipeline#nodeIdsHash collision issue. (#478) new d5666c5 HDDS-2923 Add fall-back protection for rack awareness in pipeline creation. (#516) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (43845da) \ N -- N -- N refs/heads/HDDS-1564 (d5666c5) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 18 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/buildenv/Dockerfile | 2 +- .github/workflows/post-commit.yml | 90 + .github/workflows/pr.yml | 90 + hadoop-hdds/client/pom.xml | 8 + .../apache/hadoop/hdds/conf/RatisClientConfig.java | 84 + .../apache/hadoop/hdds/conf/RatisGrpcConfig.java | 35 +- .../org/apache/hadoop/hdds/conf/package-info.java | 11 +- .../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 25 +- .../hadoop/hdds/scm/XceiverClientManager.java | 13 - .../apache/hadoop/hdds/scm/XceiverClientRatis.java | 35 +- .../hdds/scm/client/ContainerOperationClient.java | 8 +- .../hadoop/hdds/scm/client/HddsClientUtils.java | 4 +- .../hadoop/hdds/scm/storage/BlockInputStream.java | 34 +- .../hadoop/hdds/scm/storage/BlockOutputStream.java | 6 +- .../hadoop/hdds/scm/storage/CommitWatcher.java | 8 +- .../hdds/scm/storage/DummyBlockInputStream.java | 92 + .../storage/DummyBlockInputStreamWithRetry.java | 78 + .../hdds/scm/storage/DummyChunkInputStream.java | 66 + .../hdds/scm/storage/TestBlockInputStream.java | 59 +- .../hdds/scm/storage/TestChunkInputStream.java | 52 +- hadoop-hdds/common/pom.xml | 32 + .../apache/hadoop/hdds/DFSConfigKeysLegacy.java | 100 + .../org/apache/hadoop/hdds/HddsConfigKeys.java | 9 +- .../java/org/apache/hadoop/hdds/HddsUtils.java | 71 +- .../java/org/apache/hadoop/hdds/StringUtils.java | 76 + .../hadoop/hdds/fs/AbstractSpaceUsageSource.java | 87 + .../hadoop/hdds/fs/CachingSpaceUsageSource.java | 142 ++ .../main/java/org/apache/hadoop/hdds/fs/DU.java | 136 ++ .../java/org/apache/hadoop/hdds/fs/DUFactory.java | 90 + .../hadoop/hdds/fs/DedicatedDiskSpaceUsage.java | 52 + .../hdds/fs/DedicatedDiskSpaceUsageFactory.java | 86 + .../hadoop/hdds/fs/SaveSpaceUsageToFile.java | 129 ++ .../hadoop/hdds/fs/SpaceUsageCheckFactory.java | 145 ++ .../hadoop/hdds/fs/SpaceUsageCheckParams.java | 92 + .../hadoop/hdds/fs/SpaceUsagePersistence.java | 63 + .../apache/hadoop/hdds/fs/SpaceUsageSource.java | 28 +- .../org/apache/hadoop/hdds/fs/package-info.java | 9 +- .../org/apache/hadoop/hdds/ratis/RatisHelper.java | 129 +- .../apache/hadoop/hdds/recon/ReconConfigKeys.java | 6 +- .../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 34 +- .../apache/hadoop/hdds/scm/XceiverClientSpi.java | 3 +- .../apache/hadoop/hdds/scm/client/ScmClient.java | 7 + .../hadoop/hdds/scm/protocol/LocatedContainer.java | 127 - .../hadoop/hdds/scm/protocol/ScmLocatedBlock.java | 100 - .../protocol/StorageContainerLocationProtocol.java | 10 + ...inerLocationProtocolClientSideTranslatorPB.java | 16 + .../hdds/scm/storage/ContainerProtocolCalls.java | 5 + .../hadoop/hdds/security/x509/SecurityConfig.java | 41 +- .../security/x509/certificate/utils/CRLCodec.java | 206 ++ .../java/org/apache/hadoop/hdds/utils/Cache.java | 27 +- .../hadoop/hdds/utils/MetadataKeyFilters.java | 15 +- .../hadoop/hdds/utils/ResourceLimitCache.java | 91 + .../hadoop/hdds/utils/ResourceSemaphore.java | 170 ++ .../org/apache/hadoop/hdds/utils/db/DBStore.java | 2 +- .../hadoop/hdds/utils/db/DBStoreBuilder.java | 6 +- .../org/apache/hadoop/hdds/utils/db/RDBStore.java | 12 +- .../org/apache/hadoop/hdds/utils/db/RDBTable.java | 4 +- .../apache/hadoop/hdds/utils/db/StringCodec.java | 7 +- .../apache/hadoop/hdds/utils/db/TableConfig.java | 5 +- .../org/apache/hadoop/ozone/OzoneConfigKeys.java | 68 +- .../java/org/apache/hadoop/ozone/OzoneConsts.java | 1 - .../ozone/conf/DatanodeRatisServerConfig.java | 141 ++ .../org/apache/hadoop/ozone/conf/package-info.java | 11 +- .../proto/StorageContainerLocationProtocol.proto | 12 + hadoop-hdds/common/src/main/proto/hdds.proto | 1 + .../common/src/main/resources/ozone-default.xml | 161 +- .../hadoop/hdds/fs/MockSpaceUsageCheckFactory.java | 57 + .../hadoop/hdds/fs/MockSpaceUsageCheckParams.java | 71 + .../hadoop/hdds/fs/MockSpaceUsagePersistence.java | 44 +- .../hadoop/hdds/fs/MockSpaceUsageSource.java | 74 + .../hdds/fs/TestCachingSpaceUsageSource.java | 197 ++ .../java/org/apache/hadoop/hdds/fs/TestDU.java | 112 + .../org/apache/hadoop/hdds/fs/TestDUFactory.java | 57 + .../hdds/fs/TestDedicatedDiskSpaceUsage.java | 63 + .../fs/TestDedicatedDiskSpaceUsageFactory.java | 57 + .../hadoop/hdds/fs/TestSaveSpaceUsageToFile.java | 150 ++ .../hadoop/hdds/fs/TestSpaceUsageFactory.java | 197 ++ .../apache/hadoop/hdds/ratis/TestRatisHelper.java | 122 + .../token/TestOzoneBlockTokenIdentifier.java | 5 +- .../x509/certificate/utils/TestCRLCodec.java | 290 +++ .../hadoop/hdds/utils/TestMetadataStore.java | 53 +- .../hadoop/hdds/utils/TestResourceLimitCache.java | 87 + .../hadoop/hdds/utils/TestResourceSemaphore.java | 76 + .../hadoop/hdds/utils/db/TestDBConfigFromFile.java | 7 +- .../apache/hadoop/hdds/utils/db/TestRDBStore.java | 25 +- .../hadoop/hdds/utils/db/TestRDBTableStore.java | 6 +- .../hdds/utils/db/TestTypedRDBTableStore.java | 5 +- hadoop-hdds/container-service/pom.xml | 14 + .../hadoop/ozone/HddsDatanodeHttpServer.java | 2 +- .../apache/hadoop/ozone/HddsDatanodeService.java | 46 +- .../container/common/helpers/ContainerMetrics.java | 4 +- .../common/statemachine/DatanodeStateMachine.java | 4 +- .../common/statemachine/EndpointStateMachine.java | 40 +- .../common/statemachine/SCMConnectionManager.java | 26 +- .../common/statemachine/StateContext.java | 114 +- .../common/states/datanode/InitDatanodeState.java | 9 +- .../states/endpoint/HeartbeatEndpointTask.java | 9 +- .../states/endpoint/VersionEndpointTask.java | 65 +- .../server/ratis/ContainerStateMachine.java | 63 +- .../transport/server/ratis/XceiverServerRatis.java | 69 +- .../ozone/container/common/volume/HddsVolume.java | 58 +- .../container/common/volume/HddsVolumeChecker.java | 101 +- .../ozone/container/common/volume/VolumeInfo.java | 44 +- .../ozone/container/common/volume/VolumeSet.java | 11 +- .../ozone/container/common/volume/VolumeUsage.java | 169 +- .../StorageContainerDatanodeProtocolPB.java | 8 +- .../hadoop/ozone/TestHddsDatanodeService.java | 10 +- .../hadoop/ozone/TestHddsSecureDatanodeInit.java | 28 +- .../ozone/container/common/SCMTestUtils.java | 5 + .../common/statemachine/TestStateContext.java | 117 + .../states/endpoint/TestHeartbeatEndpointTask.java | 8 + .../container/common/volume/TestHddsVolume.java | 59 +- .../common/volume/TestHddsVolumeChecker.java | 7 +- .../volume/TestRoundRobinVolumeChoosingPolicy.java | 90 +- .../container/common/volume/TestVolumeSet.java | 4 +- .../common/volume/TestVolumeSetDiskChecks.java | 10 +- .../src/test/resources/ozone-site.xml | 30 + hadoop-hdds/docs/content/concept/_index.zh.md | 8 +- .../S3.zh.md => gdpr/GDPR in Ozone.zh.md} | 21 +- hadoop-hdds/docs/content/gdpr/_index.zh.md | 34 + hadoop-hdds/docs/content/interface/JavaApi.zh.md | 134 +- hadoop-hdds/docs/content/interface/OzoneFS.zh.md | 140 ++ hadoop-hdds/docs/content/interface/S3.zh.md | 130 +- .../docs/content/{shell => interface}/_index.zh.md | 8 +- hadoop-hdds/docs/content/shell/BucketCommands.md | 1 + .../docs/content/shell/BucketCommands.zh.md | 98 + hadoop-hdds/docs/content/shell/Format.md | 4 +- hadoop-hdds/docs/content/shell/Format.zh.md | 65 + hadoop-hdds/docs/content/shell/KeyCommands.md | 4 +- hadoop-hdds/docs/content/shell/KeyCommands.zh.md | 138 ++ .../docs/content/shell/VolumeCommands.zh.md | 107 + hadoop-hdds/docs/content/shell/_index.zh.md | 7 +- hadoop-hdds/docs/content/start/FromSource.zh.md | 59 + hadoop-hdds/docs/content/start/Kubernetes.zh.md | 54 + hadoop-hdds/docs/content/start/Minikube.zh.md | 70 + hadoop-hdds/docs/content/start/OnPrem.zh.md | 154 ++ .../docs/content/start/RunningViaDocker.zh.md | 52 + .../docs/content/start/StartFromDockerHub.zh.md | 96 + hadoop-hdds/docs/content/start/_index.zh.md | 84 + hadoop-hdds/docs/dev-support/bin/generate-site.sh | 2 + hadoop-hdds/framework/pom.xml | 25 + .../org/apache/hadoop/hdds/server/ServerUtils.java | 11 +- .../hadoop/hdds/server/events/EventQueue.java | 13 +- .../hdds/server/http/AdminAuthorizedServlet.java | 37 +- .../hdds/server/{ => http}/BaseHttpServer.java | 207 +- .../FilterContainer.java} | 32 +- .../hadoop/hdds/server/http/FilterInitializer.java | 18 +- .../hadoop/hdds/server/http/HtmlQuoting.java | 235 ++ .../apache/hadoop/hdds/server/http/HttpConfig.java | 75 + .../hadoop/hdds/server/http/HttpRequestLog.java | 107 + .../hdds/server/http/HttpRequestLogAppender.java | 49 +- .../hadoop/hdds/server/http/HttpServer2.java | 1707 ++++++++++++++ .../hdds/server/{ => http}/LogStreamServlet.java | 2 +- .../hadoop/hdds/server/http/NoCacheFilter.java | 55 + .../hdds/server/{ => http}/ProfileServlet.java | 2 +- .../server/{ => http}/PrometheusMetricsSink.java | 2 +- .../hdds/server/{ => http}/PrometheusServlet.java | 2 +- .../server/{ => http}/RatisDropwizardExports.java | 2 +- .../{ => http}/RatisNameRewriteSampleBuilder.java | 2 +- .../hdds/server/http/StaticUserWebFilter.java | 157 ++ .../hadoop/hdds/server/http/package-info.java | 8 +- .../hdds/server/{ => http}/TestBaseHttpServer.java | 3 +- .../hadoop/hdds/server/http/TestHtmlQuoting.java | 96 + .../hdds/server/http/TestHttpRequestLog.java | 51 + .../server/http/TestHttpRequestLogAppender.java | 33 +- .../hdds/server/{ => http}/TestProfileServlet.java | 6 +- .../{ => http}/TestPrometheusMetricsSink.java | 2 +- .../{ => http}/TestRatisDropwizardExports.java | 2 +- .../server/{ => http}/TestRatisNameRewrite.java | 2 +- .../hadoop/hdds/server/http/package-info.java | 8 +- hadoop-hdds/pom.xml | 50 - hadoop-hdds/server-scm/pom.xml | 43 +- .../container/AbstractContainerReportHandler.java | 2 +- .../hdds/scm/container/ContainerManager.java | 2 + .../hdds/scm/container/ContainerStateManager.java | 15 +- .../IncrementalContainerReportHandler.java | 3 + .../hdds/scm/container/SCMContainerManager.java | 30 +- .../hadoop/hdds/scm/node/SCMNodeManager.java | 20 +- .../hadoop/hdds/scm/pipeline/PipelineFactory.java | 15 +- .../hadoop/hdds/scm/pipeline/PipelineManager.java | 2 + .../hdds/scm/pipeline/PipelineReportHandler.java | 38 +- .../hdds/scm/pipeline/PipelineStateManager.java | 11 +- .../hdds/scm/pipeline/RatisPipelineUtils.java | 9 +- .../hdds/scm/pipeline/SCMPipelineManager.java | 68 +- ...inerLocationProtocolServerSideTranslatorPB.java | 18 + .../hdds/scm/safemode/SCMSafeModeManager.java | 2 +- .../hadoop/hdds/scm/safemode/SafeModeManager.java | 14 +- .../hdds/scm/server/SCMClientProtocolServer.java | 9 +- .../hdds/scm/server/SCMDatanodeProtocolServer.java | 6 +- .../hadoop/hdds/scm/server/SCMStorageConfig.java | 6 + .../hdds/scm/server/StorageContainerManager.java | 3 +- .../server/StorageContainerManagerHttpServer.java | 2 +- .../scm/TestStorageContainerManagerHttpServer.java | 25 +- .../hadoop/hdds/scm/node/TestSCMNodeManager.java | 73 +- .../ozone/container/common/TestEndPoint.java | 83 +- .../apache/hadoop/ozone/client/ObjectStore.java | 5 +- .../apache/hadoop/ozone/client/OzoneBucket.java | 6 +- .../ozone/client/io/BlockOutputStreamEntry.java | 2 +- .../hadoop/ozone/client/io/KeyInputStream.java | 42 +- .../hadoop/ozone/client/io/KeyOutputStream.java | 5 - .../apache/hadoop/ozone/client/rpc/RpcClient.java | 44 +- .../org/apache/hadoop/ozone/om/OMConfigKeys.java | 7 - .../ozone/om/exceptions/OMReplayException.java} | 23 +- .../hadoop/ozone/om/helpers/OmBucketInfo.java | 84 +- .../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 100 +- .../ozone/om/helpers/OmMultipartKeyInfo.java | 110 +- .../src/main/proto/OzoneManagerProtocol.proto | 14 +- .../om/codec/TestOmMultipartKeyInfoCodec.java | 12 +- .../ozone/om/helpers/TestOmMultipartKeyInfo.java | 11 +- hadoop-ozone/dev-support/checks/integration.sh | 10 +- hadoop-ozone/dev-support/checks/unit.sh | 5 +- .../dist/src/main/compose/ozone-mr/common-config | 2 + .../src/main/compose/ozone-om-ha-s3/docker-config | 4 +- .../src/main/compose/ozone-topology/docker-config | 4 +- .../src/main/compose/ozone/docker-compose.yaml | 1 - .../dist/src/main/compose/ozone/docker-config | 4 +- hadoop-ozone/dist/src/main/compose/ozone/test.sh | 4 + .../src/main/compose/ozones3-haproxy/docker-config | 4 +- .../compose/ozonesecure-mr/docker-compose.yaml | 9 + .../src/main/compose/ozonesecure-mr/docker-config | 5 +- .../main/compose/ozonesecure/docker-compose.yaml | 10 +- .../src/main/compose/ozonesecure/docker-config | 7 +- .../dist/src/main/compose/ozonesecure/test.sh | 2 + hadoop-ozone/dist/src/main/compose/testlib.sh | 22 +- .../dist/src/main/smoketest/basic/basic.robot | 1 + .../main/smoketest/om-ratis/testOMAdminCmd.robot | 21 +- .../dist/src/main/smoketest/recon/recon-api.robot | 25 +- .../main/smoketest/security/ozone-secure-fs.robot | 5 + .../mini-chaos-tests/src/test/bin/start-chaos.sh | 16 +- .../hadoop/ozone/MiniOzoneLoadGenerator.java | 103 +- .../hadoop/ozone/TestMiniChaosOzoneCluster.java | 20 +- .../org/apache/hadoop/ozone/utils/LoadBucket.java | 224 +- .../hadoop/ozone/insight/BaseInsightPoint.java | 2 +- hadoop-ozone/integration-test/pom.xml | 132 +- .../hadoop/fs/ozone/TestOzoneFSInputStream.java | 18 +- .../hadoop/fs/ozone/TestOzoneFileSystem.java | 128 +- .../hadoop/fs/ozone/TestOzoneFsRenameDir.java | 110 - .../TestContainerStateManagerIntegration.java | 1 + .../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 16 +- .../TestRatisPipelineCreateAndDestroy.java | 2 + .../scm/pipeline/TestRatisPipelineProvider.java | 187 -- .../safemode/TestSCMSafeModeWithPipelineRules.java | 2 + .../org/apache/hadoop/ozone/MiniOzoneCluster.java | 13 + .../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 172 +- .../org/apache/hadoop/ozone/RatisTestHelper.java | 9 +- .../apache/hadoop/ozone/TestMiniOzoneCluster.java | 7 +- .../hadoop/ozone/TestOzoneConfigurationFields.java | 5 +- .../hadoop/ozone/TestStorageContainerManager.java | 2 + .../ozone/client/rpc/Test2WayCommitInRatis.java | 6 +- .../ozone/client/rpc/TestBlockOutputStream.java | 1 - .../rpc/TestBlockOutputStreamWithFailures.java | 3 +- .../rpc/TestCloseContainerHandlingByClient.java | 2 +- .../hadoop/ozone/client/rpc/TestCommitWatcher.java | 7 +- .../rpc/TestContainerReplicationEndToEnd.java | 12 +- .../client/rpc/TestContainerStateMachine.java | 3 +- .../rpc/TestContainerStateMachineFailures.java | 2 + .../client/rpc/TestDeleteWithSlowFollower.java | 11 +- .../client/rpc/TestFailureHandlingByClient.java | 4 +- .../ozone/client/rpc/TestKeyInputStream.java | 1 - .../rpc/TestMultiBlockWritesWithDnFailures.java | 4 +- .../client/rpc/TestOzoneAtRestEncryption.java | 10 +- .../rpc/TestOzoneClientRetriesOnException.java | 3 +- .../client/rpc/TestOzoneRpcClientAbstract.java | 32 +- .../client/rpc/TestOzoneRpcClientWithRatis.java | 2 + .../hadoop/ozone/client/rpc/TestReadRetries.java | 2 +- .../ozone/client/rpc/TestSecureOzoneRpcClient.java | 4 +- .../ozone/client/rpc/TestWatchForCommit.java | 62 +- .../ozone/container/TestContainerReplication.java | 2 + .../commandhandler/TestBlockDeletion.java | 2 + .../TestCloseContainerByPipeline.java | 2 + .../transport/server/ratis/TestCSMMetrics.java | 2 + .../container/metrics/TestContainerMetrics.java | 4 +- .../container/ozoneimpl/TestOzoneContainer.java | 2 + .../server/TestSecureContainerServer.java | 2 + .../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 4 +- .../hadoop/ozone/freon/TestDataValidate.java | 2 - .../ozone/freon/TestFreonWithPipelineDestroy.java | 2 + .../ozone/freon/TestOzoneClientKeyGenerator.java | 93 + .../hadoop/ozone/freon/TestRandomKeyGenerator.java | 2 - .../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 175 +- .../org/apache/hadoop/ozone/om/TestKeyPurging.java | 3 +- .../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 7 +- .../hadoop/ozone/om/TestOMRatisSnapshots.java | 2 + .../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 4 +- .../hadoop/ozone/om/TestOzoneManagerRestart.java | 2 + .../ozone/om/TestOzoneManagerRocksDBLogging.java | 46 +- .../apache/hadoop/ozone/om/TestScmSafeMode.java | 2 + .../hadoop/ozone/ozShell/TestOzoneShellHA.java | 63 + .../org/apache/hadoop/ozone/recon/TestRecon.java | 384 ++++ .../scm/TestGetCommittedBlockLengthAndPutKey.java | 4 +- .../TestSCMContainerPlacementPolicyMetrics.java | 5 +- .../hadoop/ozone/scm/node/TestQueryNode.java | 2 + .../ozone/scm/pipeline/TestSCMPipelineMetrics.java | 2 + .../resources/{hdfs-site.xml => mapred-site.xml} | 0 .../src/test/resources/ozone-site.xml | 29 + .../resources/{hdfs-site.xml => yarn-site.xml} | 0 .../apache/hadoop/ozone/om/KeyDeletingService.java | 46 +- .../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 14 +- .../hadoop/ozone/om/OMDBCheckpointServlet.java | 3 +- .../java/org/apache/hadoop/ozone/om/OMMetrics.java | 15 + .../hadoop/ozone/om/OmMetadataManagerImpl.java | 41 +- .../org/apache/hadoop/ozone/om/OzoneManager.java | 46 +- .../hadoop/ozone/om/OzoneManagerHttpServer.java | 2 +- .../apache/hadoop/ozone/om/ha/OMNodeDetails.java | 3 +- .../ozone/om/ratis/OzoneManagerRatisServer.java | 13 - .../om/ratis/utils/OzoneManagerRatisUtils.java | 2 +- .../om/request/bucket/OMBucketCreateRequest.java | 45 +- .../om/request/bucket/OMBucketDeleteRequest.java | 39 +- .../request/bucket/OMBucketSetPropertyRequest.java | 67 +- .../om/request/bucket/acl/OMBucketAclRequest.java | 18 + .../request/bucket/acl/OMBucketAddAclRequest.java | 5 +- .../bucket/acl/OMBucketRemoveAclRequest.java | 5 +- .../request/bucket/acl/OMBucketSetAclRequest.java | 5 +- .../om/request/file/OMDirectoryCreateRequest.java | 6 +- .../ozone/om/request/file/OMFileCreateRequest.java | 156 +- .../om/request/key/OMAllocateBlockRequest.java | 105 +- .../ozone/om/request/key/OMKeyCommitRequest.java | 5 +- .../ozone/om/request/key/OMKeyCreateRequest.java | 141 +- .../ozone/om/request/key/OMKeyDeleteRequest.java | 86 +- .../ozone/om/request/key/OMKeyPurgeRequest.java | 127 +- .../ozone/om/request/key/OMKeyRenameRequest.java | 188 +- .../hadoop/ozone/om/request/key/OMKeyRequest.java | 188 +- .../request/s3/bucket/S3BucketCreateRequest.java | 156 +- .../request/s3/bucket/S3BucketDeleteRequest.java | 42 +- .../S3InitiateMultipartUploadRequest.java | 17 +- .../multipart/S3MultipartUploadAbortRequest.java | 1 + .../S3MultipartUploadCommitPartRequest.java | 5 + .../S3MultipartUploadCompleteRequest.java | 19 +- .../om/response/bucket/OMBucketCreateResponse.java | 29 +- .../om/response/bucket/OMBucketDeleteResponse.java | 29 +- .../bucket/OMBucketSetPropertyResponse.java | 29 +- .../response/bucket/acl/OMBucketAclResponse.java | 21 +- .../om/response/file/OMFileCreateResponse.java | 13 +- .../om/response/key/OMAllocateBlockResponse.java | 31 +- .../ozone/om/response/key/OMKeyCreateResponse.java | 30 +- .../ozone/om/response/key/OMKeyDeleteResponse.java | 13 +- .../ozone/om/response/key/OMKeyPurgeResponse.java | 22 +- .../ozone/om/response/key/OMKeyRenameResponse.java | 81 +- .../response/s3/bucket/S3BucketCreateResponse.java | 41 +- .../response/s3/bucket/S3BucketDeleteResponse.java | 26 +- .../om/snapshot/OzoneManagerSnapshotProvider.java | 43 +- .../hadoop/ozone/web/ozShell/OzoneAddress.java | 37 + .../ozone/web/ozShell/s3/GetS3SecretHandler.java | 10 +- .../web/ozShell/volume/ListVolumeHandler.java | 3 +- .../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 11 +- .../hadoop/ozone/om/TestOmMetadataManager.java | 34 + .../ozone/om/TestOzoneManagerHttpServer.java | 8 +- ...TestOzoneManagerDoubleBufferWithOMResponse.java | 11 +- .../ozone/om/request/TestOMRequestUtils.java | 129 +- .../ozone/om/request/bucket/TestBucketRequest.java | 1 + .../request/bucket/TestOMBucketCreateRequest.java | 48 +- .../request/bucket/TestOMBucketDeleteRequest.java | 46 +- .../bucket/TestOMBucketSetPropertyRequest.java | 34 +- .../om/request/file/TestOMFileCreateRequest.java | 42 +- .../om/request/key/TestOMKeyCreateRequest.java | 48 +- .../om/request/key/TestOMKeyDeleteRequest.java | 44 +- .../key/TestOMKeyPurgeRequestAndResponse.java | 166 +- .../om/request/key/TestOMKeyRenameRequest.java | 106 +- .../ozone/om/request/key/TestOMKeyRequest.java | 1 + .../s3/bucket/TestS3BucketCreateRequest.java | 28 +- .../s3/bucket/TestS3BucketDeleteRequest.java | 36 + .../om/request/s3/bucket/TestS3BucketRequest.java | 2 +- .../ozone/om/response/TestOMResponseUtils.java | 6 +- .../bucket/TestOMBucketCreateResponse.java | 5 +- .../bucket/TestOMBucketDeleteResponse.java | 17 +- .../bucket/TestOMBucketSetPropertyResponse.java | 5 +- .../response/key/TestOMAllocateBlockResponse.java | 6 +- .../om/response/key/TestOMKeyCreateResponse.java | 6 +- .../om/response/key/TestOMKeyDeleteResponse.java | 12 +- .../om/response/key/TestOMKeyRenameResponse.java | 14 +- .../s3/bucket/TestS3BucketDeleteResponse.java | 2 +- .../s3/multipart/TestS3MultipartResponse.java | 11 +- .../hadoop/fs/ozone/BasicOzoneFileSystem.java | 21 +- .../fs/ozone/CapableOzoneFSInputStream.java} | 35 +- .../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 19 +- .../apache/hadoop/fs/ozone/OzoneFileSystem.java | 6 + .../hadoop/fs/ozone/OzoneStreamCapabilities.java} | 21 +- .../hadoop/fs/ozone/TestOzoneFSInputStream.java | 151 ++ .../ozone/contract/TestOzoneFSInputStreamUnit.java | 101 - .../hadoop/ozone/recon/ConfigurationProvider.java | 17 +- .../apache/hadoop/ozone/recon/ReconConstants.java | 11 +- .../hadoop/ozone/recon/ReconControllerModule.java | 31 +- .../apache/hadoop/ozone/recon/ReconHttpServer.java | 6 +- .../org/apache/hadoop/ozone/recon/ReconServer.java | 79 +- .../hadoop/ozone/recon/ReconServerConfigKeys.java | 5 + .../org/apache/hadoop/ozone/recon/ReconUtils.java | 5 + .../ozone/recon/api/ContainerKeyService.java | 8 +- .../recon/recovery/ReconOmMetadataManagerImpl.java | 4 +- .../ozone/recon/scm/ReconContainerManager.java | 86 + .../recon/scm/ReconDatanodeProtocolServer.java | 31 +- .../ReconIncrementalContainerReportHandler.java | 70 +- .../ozone/recon/scm/ReconPipelineFactory.java | 72 + .../ozone/recon/scm/ReconPipelineManager.java | 139 ++ .../recon/scm/ReconPipelineReportHandler.java | 90 + .../ReconSafeModeManager.java} | 16 +- ...conNodeManager.java => ReconStorageConfig.java} | 23 +- .../recon/scm/ReconStorageContainerManager.java | 134 -- .../scm/ReconStorageContainerManagerFacade.java | 234 ++ .../recon/spi/ContainerDBServiceProvider.java | 5 + .../recon/spi/StorageContainerServiceProvider.java | 31 + .../spi/impl/ContainerDBServiceProviderImpl.java | 30 +- .../spi/impl/OzoneManagerServiceProviderImpl.java | 18 +- .../recon/spi/impl/ReconContainerDBProvider.java | 22 +- .../impl/StorageContainerServiceProviderImpl.java | 63 + .../ozone/recon/tasks/ReconTaskController.java | 7 +- .../ozone/recon/tasks/ReconTaskControllerImpl.java | 11 +- .../webapps/recon/ozone-recon-web/LICENSE | 1422 ++++-------- .../webapps/recon/ozone-recon-web/api/db.json | 380 ++- .../webapps/recon/ozone-recon-web/api/routes.json | 3 +- .../webapps/recon/ozone-recon-web/package.json | 2 + .../webapps/recon/ozone-recon-web/src/App.less | 31 + .../src/components/NavBar/NavBar.less | 21 +- .../src/components/OverviewCard/OverviewCard.less | 4 + .../src/components/OverviewCard/OverviewCard.tsx | 13 +- .../src/constants/breadcrumbs.constants.tsx | 3 +- .../webapps/recon/ozone-recon-web/src/routes.tsx | 10 + .../Overview/Overview.less => utils/common.tsx} | 4 +- .../Datanodes/Datanodes.less} | 31 +- .../src/views/Datanodes/Datanodes.tsx | 178 +- .../MissingContainers.less} | 4 - .../views/MissingContainers/MissingContainers.tsx | 218 ++ .../src/views/Overview/Overview.less | 9 + .../src/views/Overview/Overview.tsx | 43 +- .../Overview.less => Pipelines/Pipelines.less} | 8 +- .../src/views/Pipelines/Pipelines.tsx | 158 ++ .../webapps/recon/ozone-recon-web/yarn.lock | 2432 +++++++++----------- .../ozone/recon/AbstractOMMetadataManagerTest.java | 5 +- .../scm/AbstractReconContainerManagerTest.java | 84 + .../ozone/recon/scm/TestReconContainerManager.java | 81 + ...TestReconIncrementalContainerReportHandler.java | 121 + .../ozone/recon/scm/TestReconPipelineManager.java | 173 ++ .../recon/scm/TestReconPipelineReportHandler.java | 101 + .../TestStorageContainerServiceProviderImpl.java | 93 + .../recon/tasks/TestReconTaskControllerImpl.java | 1 + .../java/org/apache/hadoop/ozone/s3/Gateway.java | 1 + .../hadoop/ozone/s3/S3GatewayHttpServer.java | 2 +- .../ozone/admin/om/GetServiceRolesSubcommand.java | 9 +- .../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 17 +- .../hadoop/ozone/freon/DatanodeChunkGenerator.java | 2 +- .../apache/hadoop/ozone/freon/FreonHttpServer.java | 2 +- .../org/apache/hadoop/ozone/freon/ProgressBar.java | 44 +- .../apache/hadoop/ozone/freon/TestProgressBar.java | 3 +- pom.xml | 4 +- 443 files changed, 18394 insertions(+), 6186 deletions(-) create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/conf/RatisClientConfig.java copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ConfigurationProvider.java => hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/conf/RatisGrpcConfig.java (51%) copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/Overview/Overview.less => hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/conf/package-info.java (88%) create mode 100644 hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/DummyBlockInputStream.java create mode 100644 hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/DummyBlockInputStreamWithRetry.java create mode 100644 hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/DummyChunkInputStream.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/DFSConfigKeysLegacy.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/AbstractSpaceUsageSource.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/CachingSpaceUsageSource.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DU.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DUFactory.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DedicatedDiskSpaceUsage.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DedicatedDiskSpaceUsageFactory.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SaveSpaceUsageToFile.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckFactory.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckParams.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsagePersistence.java copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageSource.java (58%) copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/Overview/Overview.less => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/package-info.java (91%) delete mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/LocatedContainer.java delete mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmLocatedBlock.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CRLCodec.java copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/constants/breadcrumbs.constants.tsx => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/Cache.java (74%) create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/ResourceLimitCache.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/ResourceSemaphore.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/conf/DatanodeRatisServerConfig.java copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/Overview/Overview.less => hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/conf/package-info.java (88%) create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/MockSpaceUsageCheckFactory.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/MockSpaceUsageCheckParams.java copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/MockSpaceUsagePersistence.java (50%) create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/MockSpaceUsageSource.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestCachingSpaceUsageSource.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestDU.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestDUFactory.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestDedicatedDiskSpaceUsage.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestDedicatedDiskSpaceUsageFactory.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestSaveSpaceUsageToFile.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestSpaceUsageFactory.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestRatisHelper.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/TestCRLCodec.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestResourceLimitCache.java create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/TestResourceSemaphore.java create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/TestStateContext.java create mode 100644 hadoop-hdds/container-service/src/test/resources/ozone-site.xml copy hadoop-hdds/docs/content/{interface/S3.zh.md => gdpr/GDPR in Ozone.zh.md} (52%) create mode 100644 hadoop-hdds/docs/content/gdpr/_index.zh.md create mode 100644 hadoop-hdds/docs/content/interface/OzoneFS.zh.md copy hadoop-hdds/docs/content/{shell => interface}/_index.zh.md (80%) create mode 100644 hadoop-hdds/docs/content/shell/BucketCommands.zh.md create mode 100644 hadoop-hdds/docs/content/shell/Format.zh.md create mode 100644 hadoop-hdds/docs/content/shell/KeyCommands.zh.md create mode 100644 hadoop-hdds/docs/content/shell/VolumeCommands.zh.md create mode 100644 hadoop-hdds/docs/content/start/FromSource.zh.md create mode 100644 hadoop-hdds/docs/content/start/Kubernetes.zh.md create mode 100644 hadoop-hdds/docs/content/start/Minikube.zh.md create mode 100644 hadoop-hdds/docs/content/start/OnPrem.zh.md create mode 100644 hadoop-hdds/docs/content/start/RunningViaDocker.zh.md create mode 100644 hadoop-hdds/docs/content/start/StartFromDockerHub.zh.md create mode 100644 hadoop-hdds/docs/content/start/_index.zh.md copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/AdminAuthorizedServlet.java (52%) rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/BaseHttpServer.java (51%) copy hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{RatisDropwizardExports.java => http/FilterContainer.java} (54%) copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java => hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/FilterInitializer.java (66%) create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HtmlQuoting.java create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpConfig.java create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpRequestLog.java copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpRequestLogAppender.java (51%) create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/LogStreamServlet.java (97%) create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/NoCacheFilter.java rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/ProfileServlet.java (99%) rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/PrometheusMetricsSink.java (99%) rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/PrometheusServlet.java (97%) rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/RatisDropwizardExports.java (96%) rename hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/{ => http}/RatisNameRewriteSampleBuilder.java (98%) create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/StaticUserWebFilter.java copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/Overview/Overview.less => hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/package-info.java (86%) rename hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/{ => http}/TestBaseHttpServer.java (98%) create mode 100644 hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHtmlQuoting.java create mode 100644 hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLog.java copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLogAppender.java (56%) rename hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/{ => http}/TestProfileServlet.java (90%) rename hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/{ => http}/TestPrometheusMetricsSink.java (99%) rename hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/{ => http}/TestRatisDropwizardExports.java (98%) rename hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/{ => http}/TestRatisNameRewrite.java (98%) copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/Overview/Overview.less => hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/package-info.java (88%) copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java => hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeManager.java (78%) copy hadoop-ozone/{recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java => common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMReplayException.java} (59%) copy hadoop-hdds/docs/dev-support/bin/generate-site.sh => hadoop-ozone/dist/src/main/smoketest/om-ratis/testOMAdminCmd.robot (65%) mode change 100755 => 100644 copy hadoop-hdds/docs/dev-support/bin/generate-site.sh => hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot (57%) mode change 100755 => 100644 delete mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFsRenameDir.java delete mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOzoneClientKeyGenerator.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestRecon.java copy hadoop-ozone/integration-test/src/test/resources/{hdfs-site.xml => mapred-site.xml} (100%) create mode 100644 hadoop-ozone/integration-test/src/test/resources/ozone-site.xml copy hadoop-ozone/integration-test/src/test/resources/{hdfs-site.xml => yarn-site.xml} (100%) copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => ozonefs/src/main/java/org/apache/hadoop/fs/ozone/CapableOzoneFSInputStream.java} (56%) copy hadoop-ozone/{recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java => ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneStreamCapabilities.java} (61%) create mode 100644 hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSInputStream.java delete mode 100644 hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/contract/TestOzoneFSInputStreamUnit.java create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconContainerManager.java copy hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java => hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconIncrementalContainerReportHandler.java (55%) create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconPipelineFactory.java create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconPipelineManager.java create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconPipelineReportHandler.java copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/{spi/StorageContainerServiceProvider.java => scm/ReconSafeModeManager.java} (73%) rename hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/{ReconNodeManager.java => ReconStorageConfig.java} (60%) delete mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManager.java create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/{views/Overview/Overview.less => utils/common.tsx} (88%) copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/{components/OverviewCard/OverviewCard.less => views/Datanodes/Datanodes.less} (71%) copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/{Overview/Overview.less => MissingContainers/MissingContainers.less} (94%) create mode 100644 hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/MissingContainers/MissingContainers.tsx copy hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/{Overview/Overview.less => Pipelines/Pipelines.less} (94%) create mode 100644 hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/Pipelines/Pipelines.tsx create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/AbstractReconContainerManagerTest.java create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconContainerManager.java create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconIncrementalContainerReportHandler.java create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconPipelineManager.java create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconPipelineReportHandler.java create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/spi/impl/TestStorageContainerServiceProviderImpl.java --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org For additional commands, e-mail: ozone-commits-help@hadoop.apache.org