Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C7DB18F2A for ; Tue, 8 Mar 2016 19:06:14 +0000 (UTC) Received: (qmail 96447 invoked by uid 500); 8 Mar 2016 19:06:04 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 96095 invoked by uid 500); 8 Mar 2016 19:06:04 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 94683 invoked by uid 99); 8 Mar 2016 19:06:03 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2016 19:06:03 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 302B4E040F; Tue, 8 Mar 2016 19:06:03 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: asuresh@apache.org To: common-commits@hadoop.apache.org Date: Tue, 08 Mar 2016 19:06:21 -0000 Message-Id: <5d060cace2824c8f9c1a5ab3eaf56fb2@git.apache.org> In-Reply-To: <225728c44e6f44489a1a42dfa3fd1883@git.apache.org> References: <225728c44e6f44489a1a42dfa3fd1883@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [20/50] [abbrv] hadoop git commit: HADOOP-11792. Remove all of the CHANGES.txt files. http://git-wip-us.apache.org/repos/asf/hadoop/blob/1bc45461/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt deleted file mode 100644 index 0ddecaf..0000000 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ /dev/null @@ -1,14611 +0,0 @@ -Hadoop HDFS Change Log - -Trunk (Unreleased) - - INCOMPATIBLE CHANGES - - HDFS-3034. Remove the deprecated DFSOutputStream.sync() method. (szetszwo) - - HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from - DatanodeProtocolProtos. (Tao Luo via shv) - - HDFS-5570. Deprecate hftp / hsftp and replace them with webhdfs / swebhdfs. - (wheat9) - - HDFS-2538. option to disable fsck dots (Mohammad Kamrul Islam via aw) - - HDFS-7302. Remove "downgrade" from "namenode -rollingUpgrade" startup - option since it may incorrectly finalize an ongoing rolling upgrade. - (Kai Sasaki via szetszwo) - - HDFS-7985. WebHDFS should be always enabled. (Li Lu via wheat9) - - HDFS-7281. Missing block is marked as corrupted block (Ming Ma via - Yongjun Zhang) - - HDFS-8349. Remove .xml and documentation references to dfs.webhdfs.enabled. - (Ray Chiang via aajisaka) - - HDFS-8241. Remove unused NameNode startup option -finalize. - (Brahma Reddy Battula via aajisaka) - - HDFS-8135. Remove the deprecated FSConstants class. (Li Lu via wheat9) - - HDFS-8591. Remove support for deprecated configuration key - dfs.namenode.decommission.nodes.per.interval. (wang) - - HDFS-8895. Remove deprecated BlockStorageLocation APIs. (wang) - - HDFS-8981. Adding revision to data node jmx getVersion() method. (Siqi Li - via mingma) - - HDFS-9085. Show renewer information in DelegationTokenIdentifier#toString. - (Zhihai Xu via cnauroth) - - HDFS-9278. Fix preferredBlockSize typo in OIV XML output. (Nicole Pazmany via wang) - - HDFS-5165. Remove the TotalFiles metrics. (Akira Ajisaka via wheat9) - - NEW FEATURES - - HDFS-3125. Add JournalService to enable Journal Daemon. (suresh) - - HDFS-6440. Support more than 2 NameNodes. (Jesse Yates via atm) - - HDFS-9057. allow/disallow snapshots via webhdfs - (Bramma Reddy Battula via vinayakumarb) - - HDFS-9525. hadoop utilities need to support provided delegation - tokens (HeeSoo Kim via aw) - - HDFS-9804. Allow long-running Balancer to login with keytab. - (Xiao Chen via zhz) - - IMPROVEMENTS - - HDFS-9653. Added blocks pending deletion report to dfsadmin. - (Weiwei Yang via eyang) - - HDFS-4665. Move TestNetworkTopologyWithNodeGroup to common. - (Junping Du via llu) - - HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants -> - HdfsConstants. (Harsh J Chouraria via atm) - - HDFS-2197. Refactor RPC call implementations out of NameNode class (todd) - - HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh) - - HDFS-2857. Cleanup BlockInfo class. (suresh) - - HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal Lee - via jitendra) - - HDFS-2878. Fix TestBlockRecovery and move it back into main test directory. - (todd) - - HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. - (Brandon Li via jitendra) - - HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using - NetUtils. (Hari Mankude via suresh) - - HDFS-3002. TestNameNodeMetrics need not wait for metrics update. - (suresh) - - HDFS-3016. Security in unit tests. (Jaimin Jetly via jitendra) - - HDFS-3030. Remove getProtocolVersion and getProtocolSignature from translators. - (jitendra) - - HDFS-3111. Missing license headers in trunk. (umamahesh) - - HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in - the config description for the smaller clusters. (szetszwo via umamahesh) - - HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi - via todd) - - HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm) - - HDFS-3178. Add states and state handler for journal synchronization in - JournalService. (szetszwo) - - HDFS-3273. Refactor BackupImage and FSEditLog, and rename - JournalListener.rollLogs(..) to startLogSegment(..). (szetszwo) - - HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawUsed() - and getRawCapacity() from DistributedFileSystem. (Arpit Gupta via szetszwo) - - HADOOP-8285. HDFS changes for Use ProtoBuf for RpcPayLoadHeader. (sanjay - radia) - - HDFS-2743. Streamline usage of bookkeeper journal manager. - (Ivan Kelly via umamahesh) - - HDFS-3293. Add toString(), equals(..) and hashCode() to JournalInfo. - (Hari Mankude via szetszwo) - - HDFS-3197. Incorrect class comments in a few tests. (Andy Isaacson via eli) - - HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() - after HDFS-3462 (harsh) - - HDFS-3478. Test quotas with Long.Max_Value. (Sujay Rau via eli) - - HDFS-3630 Modify TestPersistBlocks to use both flush and hflush (sanjay) - - HDFS-3768. Exception in TestJettyHelper is incorrect. - (Eli Reisman via jghoman) - - HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. (harsh) - - HDFS-2127. Add a test that ensure AccessControlExceptions contain - a full path. (Stephen Chu via eli) - - HDFS-3995. Use DFSTestUtil.createFile() for file creation and - writing in test cases. (Jing Zhao via suresh) - - HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields - Block Pool Used, Block Pool Used(%) and Failed Volumes. - (Brahma Reddy Battula via suresh) - - HDFS-4052. BlockManager#invalidateWork should print log outside the lock. - (Jing Zhao via suresh) - - HDFS-3358. Specify explicitly that the NN UI status total is talking - of persistent objects on heap. (harsh) - - HDFS-4904. Remove JournalService. (Arpit Agarwal via cnauroth) - - HDFS-5041. Add the time of last heartbeat to dead server Web UI (Shinichi - Yamashita via brandonli) - - HDFS-5721. sharedEditsImage in Namenode#initializeSharedEdits() should be - closed before method returns. (Ted Yu via junping_du) - - HDFS-6228. comments typo fix for FsDatasetImpl.java (zhaoyunjiong via umamahesh) - - HDFS-6246. Remove 'dfs.support.append' flag from trunk code. (umamahesh) - - HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable - directory. (Jing Zhao via wheat9) - - HDFS-7591. hdfs classpath command should support same options as hadoop - classpath (Varun Saxena via Arpit Agarwal) - - HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu - Yao via aw) - - HDFS-7322. deprecate sbin/hadoop-daemon.sh (aw) - - HDFS-7668. Convert site documentation from apt to markdown (Masatake - Iwasaki via aw) - - HDFS-7460. Rewrite httpfs to use new shell framework (John Smith via aw) - - HDFS-6353. Check and make checkpoint before stopping the NameNode. (jing9) - - HDFS-5033. Bad error message for fs -put/copyFromLocal if user - doesn't have permissions to read the source (Darrell Taylor via aw) - - HDFS-9182. Cleanup the findbugs and other issues after HDFS EC merged to trunk. - (umamahesh) - - HDFS-8632. Add InterfaceAudience annotation to the erasure coding classes. - (Rakesh R via wang) - - HDFS-9070. Allow fsck display pending replica location information for - being-written blocks. (GAO Rui via jing9) - - HDFS-9261. Erasure Coding: Skip encoding the data cells if all the parity data - streamers are failed for the current block group. (Rakesh R via umamahesh) - - HDFS-9323. Randomize the DFSStripedOutputStreamWithFailure tests. (szetszwo) - - HDFS-8777. Erasure Coding: add tests for taking snapshots on EC files. - (Rakesh R via zhz) - - HADOOP-11684. S3a to use thread pool that blocks clients. - (Thomas Demoor and Aaron Fabbri via lei) - - HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. - (Surendra Singh Lilhore via xyao) - - HDFS-8968. Erasure coding: a comprehensive I/O throughput benchmark tool. - (Rui Li via zhz) - - HDFS-8831. Trash Support for deletion in HDFS encryption zone. (xyao) - - HDFS-9582. TestLeaseRecoveryStriped file missing Apache License header - and not well formatted. (umamahesh) - - HDFS-9494. Parallel optimization of DFSStripedOutputStream#flushAllInternals. - (Gao Rui via jing9) - - HDFS-9260. Improve the performance and GC friendliness of NameNode startup - and full block reports (Staffan Friberg via cmccabe) - - HDFS-9795. OIV Delimited should show which files are ACL-enabled (lei) - - HDFS-9829. Erasure Coding: Improve few exception handling logic of - ErasureCodingWorker. (Rakesh R via jing9) - - HDFS-9838. Refactor the excessReplicateMap to a class. (szetszwo) - - HDFS-9733. Refactor DFSClient#getFileChecksum and DataXceiver#blockChecksum - (Kai Zheng via umamahesh) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9635 Fix potential Stack Overflow in DomainSocket.c (V. Karthik Kumar - via cmccabe) - - HDFS-2299. TestOfflineEditsViewer is failing on trunk. (Uma Maheswara Rao G - via atm) - HDFS-2310. TestBackupNode fails since HADOOP-7524 went in. - (Ivan Kelly via todd) - - HDFS-2313. Rat excludes has a typo for excluding editsStored files. (atm) - - HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd) - - HDFS-46. Change default namespace quota of root directory from - Integer.MAX_VALUE to Long.MAX_VALUE. (Uma Maheswara Rao G via szetszwo) - - HDFS-2373. Commands using WebHDFS and hftp print unnecessary debug - info on the console with security enabled. (Arpit Gupta via suresh) - - HDFS-2776. Missing interface annotation on JournalSet. - (Brandon Li via jitendra) - - HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is - racy. (atm) - - HDFS-3116. Typo in fetchdt error message. (AOE Takashi via atm) - - HDFS-3126. Journal stream from Namenode to BackupNode needs to have - timeout. (Hari Mankude via suresh) - - HDFS-3121. Add HDFS tests for HADOOP-8014 change. (John George via - suresh) - - HDFS-3119. Overreplicated block is not deleted even after the replication - factor is reduced after sync follwed by closing that file. (Ashish Singhi - via umamahesh) - - HDFS-3265. PowerPc Build error. (Kumar Ravi via mattf) - - HDFS-2312. FSNamesystem javadoc incorrectly says its for DNs. (harsh) - - HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase. - (Brandon Li via atm) - - HDFS-3462. TestDFSClientRetries.busyTest() should restore default - xceiver count in the config. (Madhukara Phatak via harsh) - - HDFS-3550. Fix raid javadoc warnings. (Jason Lowe via daryn) - - HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) - - HDFS-3614. Revert unused MiniDFSCluster constructor from HDFS-3049. - (acmurthy via eli) - - HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd) - - HDFS-3827. TestHASafeMode#assertSafemode method should be made static. - (Jing Zhao via suresh) - - HDFS-3834. Remove unused static fields NAME, DESCRIPTION and Usage from - Command. (Jing Zhao via suresh) - - HDFS-2434. TestNameNodeMetrics.testCorruptBlock fails intermittently. - (Jing Zhao via suresh) - - HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to - ReplicaAlreadyExistsException. (Jing Zhao via suresh) - - HDFS-4115. TestHDFSCLI.testAll fails one test due to number format. - (Trevor Robinson via suresh) - - HDFS-4105. The SPNEGO user for secondary namenode should use the web - keytab. (Arpit Gupta via jitendra) - - HDFS-4003. test-patch should build the common native libs before - running hdfs tests. (Colin Patrick McCabe via eli) - - HDFS-4260 Fix HDFS tests to set test dir to a valid HDFS path as opposed - to the local build path (Chri Nauroth via Sanjay) - - HDFS-4310. fix test org.apache.hadoop.hdfs.server.datanode. - TestStartSecureDataNode (Ivan A. Veselovsky via atm) - - HDFS-4275. MiniDFSCluster-based tests fail on Windows due to failure - to delete test namenode directory. (Chris Nauroth via suresh) - - HDFS-4338. TestNameNodeMetrics#testCorruptBlock is flaky. (Andrew Wang via - atm) - - HDFS-4391. TestDataTransferKeepalive fails when tests are executed in a - certain order. (Andrew Wang via atm) - - HDFS-4757. Update FSDirectory#inodeMap when replacing an INodeDirectory - while setting quota. (Jing Zhao via szetszwo) - - HDFS-4761. When resetting FSDirectory, the inodeMap should also be reset. - (Jing Zhao via szetszwo) - - HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with - JDK7. (Andrew Wang via atm) - - HDFS-3934. duplicative dfs_hosts entries handled wrong. (Colin Patrick - McCabe) - - HDFS-5719. FSImage#doRollback() should close prevState before return - (Ted Yu via brandonli) - - HDFS-5794. Fix the inconsistency of layout version number of - ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. (jing9) - - HDFS-6893. crypto subcommand is not sorted properly in hdfs's hadoop_usage - (David Luo via aw) - - HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal) - - HDFS-7204. balancer doesn't run as a daemon (aw) - - HDFS-7319. Remove dead link to HFTP documentation from index.xml. - (Masatake Iwasaki via wheat9) - - HDFS-7407. Minor typo in privileged pid/out/log names (aw) - - HDFS-7581. HDFS documentation needs updating post-shell rewrite (aw) - - HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM - AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) - - HDFS-7667. Various typos and improvements to HDFS Federation doc - (Charles Lamb via aw) - - HDFS-3750. API docs don't include HDFS (Jolly Chen via aw) - - HDFS-7320. The appearance of hadoop-hdfs-httpfs site docs is inconsistent - (Masatake Iwasaki via aw) - - HDFS-7670. HDFS Quota guide has typos, incomplete command lines - (Brahma Reddy Battula via aw) - - HDFS-7669. HDFS Design Doc references commands that no longer exist. - (Brahma Reddy Battula via aw) - - HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry - Sivachenko via aw) - - HDFS-7711. Fix various typos in ClusterSetup.md. - (Brahma Reddy Battula via wheat9) - - HDFS-7791. dfs count -v should be added to quota documentation (Akira - AJISAKA via aw) - - HDFS-7803. Wrong command mentioned in HDFSHighAvailabilityWithQJM - documentation (Arshad Mohammad via aw) - - HDFS-4681. TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks - fails using IBM java (Ayappan via aw) - - HDFS-8110. Remove unsupported 'hdfs namenode -rollingUpgrade downgrade' - from document. (J.Andreina via aajisaka) - - HDFS-7673. synthetic load generator docs give incorrect/incomplete commands - (Brahma Reddy Battula via aw) - - HDFS-8332. DFS client API calls should check filesystem closed - (Rakesh R via umamahesh) - - HDFS-8412. Fix the test failures in HTTPFS. (umamahesh) - - HDFS-8627. NPE thrown if unable to fetch token from Namenode - (J.Andreina via vinayakumarb) - - HDFS-8657. Update docs for mSNN. (Jesse Yates via atm) - - HDFS-8800. hdfs --daemon stop namenode corrupts logs (John Smith via aw) - - HDFS-9204. DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly - calculated. (Mingliang Liu via jing9) - - HDFS-9217. Fix broken findbugsExcludeFile.xml for hadoop-hdfs-client module. - (Mingliang Liu via aajisaka) - - HDFS-9423. Fix intermittent failure of TestEditLogTailer. - (Masatake Iwasaki via waltersu4549) - - HDFS-9580. TestComputeInvalidateWork#testDatanodeReRegistration failed - due to unexpected number of invalidate blocks. (Wei-Chiu Chuang via zhz) - - HDFS-9376. TestSeveralNameNodes fails occasionally. - (Masatake Iwasaki via cnauroth) - - HDFS-9615. Fix variable name typo in DFSConfigKeys. (Ray Chiang via - Arpit Agarwal) - - HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with - length less than the first internal block. (jing9) - - HDFS-9789. Correctly update DataNode's scheduled block size when writing - small EC file. (jing9) - - HDFS-9794. Streamer threads may leak if failure happens when closing the - striped outputstream. (jing9) - - HDFS-9818. Correctly handle EC reconstruction work caused by not enough - racks. (jing9) - - HDFS-9837. BlockManager#countNodes should be able to detect duplicated - internal blocks. (jing9) - - HDFS-9734. Refactoring of checksum failure report related codes. - (Kai Zheng via zhz) - - HDFS-9866. BlockManager#chooseExcessReplicasStriped may weaken rack fault - tolerance. (jing9) - - HDFS-9867. Missing block exception should carry locatedBlocks information. - (Mingliang Liu via jing9) - - HDFS-9876. shouldProcessOverReplicated should not count number of pending - replicas. (jing9) - - BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS - - HDFS-7347. Configurable erasure coding policy for individual files and - directories ( Zhe Zhang via vinayakumarb ) - - HDFS-7339. Representing striped block groups in NameNode with hierarchical - naming protocol ( zhz ) - - HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang) - - HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao) - - HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao) - - HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode - (Jing Zhao via Zhe Zhang) - - HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print - striped blocks (Takuya Fukudome via jing9) - - HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from - striped files (Jing Zhao) - - HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped - blocks ( Kai Sasaki via jing9 ) - - HDFS-7912. Erasure Coding: track BlockInfo instead of Block in - UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao) - - HDFS-7369. Erasure coding: distribute recovery work for striped blocks to - DataNode (Zhe Zhang) - - HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks - (GAO Rui via jing9) - - HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage - ( Hui Zheng via jing9 ) - - HDFS-7616. Add a test for BlockGroup support in FSImage. - (Takuya Fukudome via szetszwo) - - HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped - blocks in NameNode (Jing Zhao) - - HDFS-8005. Erasure Coding: simplify striped block recovery work computation - and add tests (Jing Zhao) - - HDFS-7617. Add unit tests for editlog transactions for EC - (Hui Zheng via Zhe Zhang) - - HDFS-7839. Erasure coding: implement facilities in NameNode to create and - manage EC zones (Zhe Zhang) - - HDFS-7969. Erasure coding: NameNode support for lease recovery of striped - block groups. (Zhe Zhang) - - HDFS-7782. Erasure coding: pread from files in striped layout. - (Zhe Zhang and Jing Zhao via Zhe Zhang) - - HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from - NameNode (vinayakumarb) - - HDFS-8074. Define a system-wide default EC schema. (Kai Zheng) - - HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks. - (Jing Zhao and Zhe Zhang via Jing Zhao) - - HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng) - - HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng) - - HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all - ECSchemas loaded in Namenode. (vinayakumarb) - - HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone. - (Vinayakumar B via Zhe Zhang) - - HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this - operation fails. (Rakesh R via Zhe Zhang) - - HDFS-8123. Erasure Coding: Better to move EC related proto messages to a - separate erasurecoding proto file (Rakesh R via vinayakumarb) - - HDFS-7349. Support DFS command for the EC encoding (vinayakumarb) - - HDFS-8120. Erasure coding: created util class to analyze striped block groups. - (Contributed by Zhe Zhang and Li Bo via Jing Zhao) - - HDFS-7994. Detect if resevered EC Block ID is already used during namenode - startup. (Hui Zheng via szetszwo) - - HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz). - - HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao) - - HDFS-7937. Erasure Coding: INodeFile quota computation unit tests. - (Kai Sasaki via Jing Zhao) - - HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks. - (Jing Zhao) - - HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for - making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb) - - HDFS-8181. createErasureCodingZone sets retryCache state as false always - (Uma Maheswara Rao G via vinayakumarb) - - HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error. - (szetszwo) - - HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to - create BlockReader. (szetszwo via Zhe Zhang) - - HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema - in FileSystemLinkResolver. (szetszwo via Zhe Zhang) - - HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration. - (umamahesh) - - HDFS-8156. Add/implement necessary APIs even we just have the system default - schema. (Kai Zheng via Zhe Zhang) - - HDFS-8136. Client gets and uses EC schema when reads and writes a stripping - file. (Kai Sasaki via Kai Zheng) - - HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last - stripe is at the block group boundary. (jing9) - - HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream. - (Yi Liu via jing9) - - HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause - block id conflicts (Jing Zhao via Zhe Zhang) - - HDFS-8033. Erasure coding: stateful (non-positional) read from files in - striped layout (Zhe Zhang) - - HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY - commands from standbynode if any (vinayakumarb) - - HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated - as Idempotent (vinayakumarb) - - HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open. - (Kai Sasaki via jing9) - - HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream - (stateful read). (Jing Zhao via Zhe Zhang) - - HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil. - (Zhe Zhang) - - HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of - datastreamer threads. (Rakesh R via Zhe Zhang) - - HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() - when loading editlog. (jing9) - - HDFS-7949. WebImageViewer need support file size calculation with striped - blocks. (Rakesh R via Zhe Zhang) - - HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249. - (Zhe Zhang via jing9) - - HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout. - (jing9) - - HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh) - - HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands - (Rakesh R via vinayakumarb) - - HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via - umamahesh) - - HDFS-7672. Handle write failure for stripping blocks and refactor the - existing code in DFSStripedOutputStream and StripedDataStreamer. (szetszwo) - - HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks. - (Yi Liu via Zhe Zhang) - - HDFS-8334. Erasure coding: rename DFSStripedInputStream related test - classes. (Zhe Zhang) - - HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding - (umamahesh) - - HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream. - (Yi Liu via jing9) - - HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via - jing9) - - HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread). - (Zhe Zhang) - - HDFS-8372. Erasure coding: compute storage type quotas for striped files, - to be consistent with HDFS-8327. (Zhe Zhang via jing9) - - HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be - handled properly (Rakesh R via zhz) - - HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu) - - HDFS-8195. Erasure coding: Fix file quota change when we complete/commit - the striped blocks. (Takuya Fukudome via zhz) - - HDFS-8364. Erasure coding: fix some minor bugs in EC CLI - (Walter Su via vinayakumarb) - - HDFS-8391. NN should consider current EC tasks handling count from DN while - assigning new tasks. (umamahesh) - - HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng) - - HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549) - - HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe. - (Walter Su via jing9) - - HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN. - (Yi Liu via jing9) - - HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz) - - HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue - configurable in DFSStripedOutputStream. (Li Bo) - - HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker. - (Rakesh R via waltersu4549) - - HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz). - - HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting file. - (Yi Liu via zhz). - - HDFS-8323. Bump GenerationStamp for write faliure in DFSStripedOutputStream. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-8427. Remove dataBlockNum and parityBlockNum from BlockInfoStriped. - (Kai Sasaki via jing9) - - HDFS-8186. Erasure coding: Make block placement policy for EC file configurable. - (Walter Su via zhz) - - HDFS-8294. Erasure Coding: Fix Findbug warnings present in erasure coding. - (Rakesh R via zhz) - - HDFS-8441. Erasure Coding: make condition check earlier for setReplication. - (waltersu4549) - - HDFS-7768. Change fsck to support EC files. (Takanobu Asanuma via szetszwo) - - HDFS-8382. Remove chunkSize and initialize from erasure coder. (Kai Zheng) - - HDFS-8408. Revisit and refactor ErasureCodingInfo (vinayakumarb) - - HDFS-8479. Erasure coding: fix striping related logic in FSDirWriteFileOp to - sync with HDFS-8421. (Zhe Zhang via jing9) - - HDFS-8481. Erasure coding: remove workarounds in client side stripped blocks - recovering. (zhz) - - HDFS-8336. Expose some administrative erasure coding operations to HdfsAdmin - (Uma Maheswara Rao G via vinayakumarb) - - HDFS-8444. Erasure Coding: fix cannot rename a zone dir - (Walter Su via vinayakumarb) - - HDFS-8517. Fix a decoding issue in stripped block recovering in client side. - (Kai Zheng via jing9) - - HDFS-8453. Erasure coding: properly handle start offset for internal blocks - in a block group. (Zhe Zhang via jing9) - - HDFS-7621. Erasure Coding: update the Balancer/Mover data migration logic. - (Walter Su via zhz) - - HDFS-8328. Follow-on to update decode for DataNode striped blocks - reconstruction. (yliu) - - HDFS-8319. Erasure Coding: support decoding for stateful read. - (Jing Zhao via zhz) - - HDFS-8460. Erasure Coding: stateful read result doesn't match data - occasionally because of flawed test. (Walter Su via zhz) - - HDFS-8556. Erasure Coding: Fix usage of 'createZone' (vinayakumarb) - - HDFS-8571. Fix TestErasureCodingCli test (Vinayakumar B via waltersu4549) - - HDFS-8450. Erasure Coding: Consolidate erasure coding zone related - implementation into a single class (Rakesh R via vinayakumarb) - - HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from - StripedBlockProto. (Yi Liu via jing9) - - HDFS-8559. Erasure Coding: fix non-protobuf fsimage for striped blocks. - (Jing Zhao via yliu) - - HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and - StripedBlocksFeature. (Walter Su via jing9) - - HDFS-8466. Refactor BlockInfoContiguous and fix NPE in - TestBlockInfo#testCopyConstructor() (vinayakumarb) - - HDFS-8254. Avoid assigning a leading streamer in StripedDataStreamer to - tolerate datanode failure. (Tsz Wo Nicholas Sze via jing9) - - HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped block. - (Walter Su via jing9) - - HDFS-8602. Erasure Coding: Client can't read(decode) the EC files which have - corrupt blocks. (jing9 and Kai Sasaki) - - HDFS-8567. Erasure Coding: SafeMode handles file smaller than a full stripe. - (Walter Su via jing9) - - HDFS-8253. DFSStripedOutputStream.closeThreads releases cellBuffers - multiple times. (Kai Sasaki via szetszwo) - - HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) resulting in - double Audit log entries (vinayakumarb) - - HDFS-8684. Erasure Coding: fix some block number calculation for striped - block. (yliu) - - HDFS-8461. Erasure coding: fix priority level of UnderReplicatedBlocks for - striped block. (Walter Su via jing9) - - HDFS-8719. Erasure Coding: client generates too many small packets when - writing parity data. (Li Bo via waltersu4549) - - HDFS-8563. Erasure Coding: fsck handles file smaller than a full stripe. - (Walter Su via jing9) - - HDFS-8484. Erasure coding: Two contiguous blocks occupy IDs belong to same - striped group. (Walter Su via jing9) - - HDFS-8744. Erasure Coding: the number of chunks in packet is not updated - when writing parity data. (Li Bo) - - HDFS-8669. Erasure Coding: handle missing internal block locations in - DFSStripedInputStream. (jing9) - - HDFS-8702. Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic - for striped block. (Kai Sasaki via jing9) - - HDFS-8734. Erasure Coding: fix one cell need two packets. (Walter Su via - jing9) - - HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. - (Jing Zhao via yliu) - - HDFS-8058. Erasure coding: use BlockInfo[] for both striped and contiguous - blocks in INodeFile. (Zhe Zhang and Yi Liu via zhz) - - HDFS-8787. Erasure coding: rename BlockInfoContiguousUC and BlockInfoStripedUC - to be consistent with trunk. (zhz) - - HDFS-8433. Erasure coding: set blockToken in LocatedStripedBlock.(waltersu4549) - - HDFS-8760. Erasure Coding: reuse BlockReader when reading the same block in pread. - (jing9) - - HDFS-8781. Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for - striped block. (Yi Liu via jing9) - - HDFS-8813. Erasure Coding: Client no need to decode missing parity blocks. - (Walter Su via jing9) - - HDFS-8798. Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream - re-fetch token when expired. (Walter Su via jing9) - - HDFS-8769. Erasure coding: unit test for SequentialBlockGroupIdGenerator. - (Rakesh R via waltersu4549) - - HDFS-8202. Improve end to end stirpping file test to add erasure recovering - test. (Xinwei Qin via zhz) - - HDFS-8804. Erasure Coding: use DirectBufferPool in DFSStripedInputStream for - buffer allocation. (jing9) - - HDFS-8399. Erasure Coding: unit test the behaviour of BlockManager recovery - work for the deleted blocks. (Rakesh R via zhz) - - HDFS-8857. Erasure Coding: Fix ArrayIndexOutOfBoundsException in - TestWriteStripedFileWithFailure. (Li Bo) - - HDFS-8827. Erasure Coding: Fix NPE when NameNode processes over-replicated - striped blocks. (Walter Su and Takuya Fukudome via jing9) - - HDFS-8854. Erasure coding: add ECPolicy to replace schema+cellSize in - hadoop-hdfs. (Walter Su via zhz) - - HDFS-8220. Erasure Coding: StripedDataStreamer fails to handle the - blocklocations which doesn't satisfy BlockGroupSize. (Rakesh R via zhz) - - HDFS-8838. Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream - when the data length is small. (szetszwo via waltersu4549) - - HDFS-8909. Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC - to use BlockUnderConstructionFeature. (Jing Zhao via waltersu4549) - - HDFS-8937. Erasure coding: do not throw exception when setting replication on - EC file. (Gao Rui via jing9) - - HDFS-8978. Erasure coding: fix 2 failed tests of DFSStripedOutputStream. - (Walter Su via jing9) - - HDFS-8833. Erasure coding: store EC schema and cell size in INodeFile and - eliminate notion of EC zones. (zhz) - - HDFS-8853. Erasure Coding: Provide ECSchema validation when setting EC - policy. (andreina via zhz) - - HDFS-8975. Erasure coding : Fix random failure in TestSafeModeWithStripedFile - (J.Andreina via vinayakumarb) - - HDFS-7351. Document the HDFS Erasure Coding feature. - (umamahesh and Zhe Zhang via wang) - - HDFS-8899. Erasure Coding: use threadpool for EC recovery tasks on DataNode. - (Rakesh R via zhz) - - HDFS-9088. Cleanup erasure coding documentation. (wang via zhz) - - HDFS-9097. Erasure coding: update EC command "-s" flag to "-p" when - specifying policy. (zhz) - - HDFS-9086. Rename dfs.datanode.stripedread.threshold.millis to - dfs.datanode.stripedread.timeout.millis. (wang via zhz) - - HDFS-8550. Erasure Coding: Fix FindBugs Multithreaded correctness Warning. - (Rakesh R via zhz) - - HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail to process - remaining tasks due to one invalid ECTask (umamahesh) - - HDFS-9091. Erasure Coding: Provide DistributedFilesystem API to - getAllErasureCodingPolicies. (Rakesh R via zhz) - - HDFS-8920. Erasure Coding: when recovering lost blocks, logs can be too - verbose and hurt performance. (Rui Li via Kai Zheng) - - HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size from - ErasureCodingPolicy (Vinayakumar B via zhz) - - HDFS-9040. Erasure coding: coordinate data streamers in - DFSStripedOutputStream. (jing9 and Walter Su) - - HDFS-9185. Fix null tracer in ErasureCodingWorker. (Rakesh R via jing9) - - HDFS-9180. Update excluded DataNodes in DFSStripedOutputStream based on failures - in data streamers. (jing9) - - HDFS-8557. Allow to configure RS and XOR raw coders (Kai Zheng) - - HDFS-9206. Inconsistent default value of dfs.datanode.stripedread.buffer.size. - (Walter Su via jing9) - - HDFS-9209. Erasure coding: Add apache license header in - TestFileStatusWithECPolicy.java. (Surendra Singh Lilhore via jing9) - - HDFS-8438. Erasure Coding: Allow concat striped files if they have the same - ErasureCodingPolicy. (Walter Su via jing9) - - HDFS-9275. Wait previous ErasureCodingWork to finish before schedule - another one. (Walter Su via yliu) - - HDFS-9348. Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent - file should be handled properly. (Rakesh R via umamahesh) - - HDFS-9373. Erasure coding: friendly log information for write operations - with some failed streamers. (Li Bo via zhz) - - HDFS-9173. Erasure Coding: Lease recovery for striped file. (Walter Su and - Jing Zhao via zhz) - - HDFS-9451. Clean up depreated umasks and related unit tests. - (Wei-Chiu Chuang via wheat9) - - HDFS-9575. Use byte array for internal block indices in a striped block. - (jing9 via szetszwo) - - HDFS-9659. EditLogTailerThread to Active Namenode RPC should timeout - (surendra singh lilhore via vinayakumarb) - - HDFS-9731. Erasure Coding: Rename BlockECRecoveryCommand to - BlockECReconstructionCommand. (Rakesh R via zhz) - - HDFS-9403. Erasure coding: some EC tests are missing timeout. - (Gao Rui via zhz) - - HDFS-9658. Erasure Coding: allow to use multiple EC policies in striping - related tests. (Rui Li via zhz) - - HDFS-9775. Erasure Coding : Rename BlockRecoveryWork to - BlockReconstructionWork. (Rakesh R via zhz) - - HDFS-9755. Erasure Coding: allow to use multiple EC policies in striping - related tests [Part 2]. (Rui Li via zhz) - - HDFS-9787. SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer - changed to false. (Guocui Mi via vinayakumarb) - - HDFS-9830. Remove references to hftp in ViewFs documentation in trunk. - (Wei-Chiu Chuang via aajisaka) - - HDFS-9816. Erasure Coding: allow to use multiple EC policies in striping - related tests [Part 3]. (Rui Li via zhz) - -Release 2.9.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - HDFS-9047. Retire libwebhdfs. (wheat9) - - NEW FEATURES - HDFS-9835. OIV: add ReverseXML processor which reconstructs an fsimage from - an XML file (cmccabe) - - IMPROVEMENTS - - HDFS-9267. TestDiskError should get stored replicas through - FsDatasetTestUtils. (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-9491. Tests should get the number of pending async delets via - FsDatasetTestUtils. (Tony Wu via lei) - - HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for - reusability. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-8947. NameNode, DataNode and NFS gateway to support JvmPauseMonitor as - a service. (Sunil G via Stevel) - - HDFS-9129. Move the safemode block count into BlockManager. (Mingliang Liu - via jing9) - - HDFS-9281. Change TestDeleteBlockPool to not explicitly use File to check - block pool existence. (lei) - - HDFS-9371. Code cleanup for DatanodeManager. (jing9) - - HDFS-9300. TestDirectoryScanner.testThrottle() is still a little flakey. - (Daniel Templeton via wang) - - HDFS-8477. describe dfs.ha.zkfc.port in hdfs-default.xml. - (Kanaka Kumar Avvaru via wang) - - HDFS-9350. Avoid creating temprorary strings in Block.toString() and - getBlockName() (Staffan Friberg via cmccabe) - - HDFS-9624. DataNode start slowly due to the initial DU command operations. - (Lin Yiqun via wang) - - HDFS-9576: HTrace: collect position/length information on read operations - (zhz via cmccabe) - - HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. (jing9) - - HDFS-9674. The HTrace span for OpWriteBlock should record the maxWriteToDisk - time. (cmccabe via zhz) - - HDFS-9094. Add command line option to ask NameNode reload - configuration. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9541. Add hdfsStreamBuilder API to libhdfs to support defaultBlockSizes - greater than 2 GB. (cmccabe via zhz) - - HDFS-9677. Rename generationStampV1/generationStampV2 to - legacyGenerationStamp/generationStamp. (Mingliang Liu via jing9) - - HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had - an error (Rakesh R via cmccabe) - - HDFS-9637. Tests for RollingFileSystemSink. (Daniel Templeton via kasha) - - HDFS-9780. RollingFileSystemSink doesn't work on secure clusters. - (Daniel Templeton via kasha) - - HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. - (jing9) - - HDFS-7964. Add support for async edit logging. (Daryn Sharp) - - HDFS-9534. Add CLI command to clear storage policy from a path. - (Xiaobing Zhou via Arpit Agarwal) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-9618. Fix mismatch between log level and guard in - BlockManager#computeRecoveryWorkForBlocks (iwasakims) - - HDFS-9621. getListing wrongly associates Erasure Coding policy to pre-existing - replicated files under an EC directory. (jing9) - - HDFS-9691. TestBlockManagerSafeMode#testCheckSafeMode fails intermittently. - (Mingliang Liu via aajisaka) - - HDFS-9608. Disk IO imbalance in HDFS with heterogeneous storages. - (Wei Zhou via wang) - - HDFS-9858. RollingFileSystemSink can throw an NPE on non-secure clusters. - (Daniel Templeton via kasha) - -Release 2.8.0 - UNRELEASED - - NEW FEATURES - - HDFS-7891. A block placement policy with best rack failure tolerance. - (Walter Su via szetszwo) - - HDFS-8131. Implement a space balanced block placement policy (Liu Shaohui - via kihwal) - - HDFS-8155. Support OAuth2 in WebHDFS. (jghoman) - - HDFS-9184. Logging HDFS operation's caller context into audit logs. - (Mingliang Liu via jitendra) - - HDFS-9244. Support nested encryption zones. (zhz) - - HDFS-9711. Integrate CSRF prevention filter in WebHDFS. (cnauroth) - - IMPROVEMENTS - - HDFS-9257. improve error message for "Absolute path required" in INode.java - to contain the rejected path (Marcell Szabo via harsh) - - HDFS-2390. dfsadmin -setBalancerBandwidth does not validate -ve value - (Gautam Gopalakrishnan via harsh) - - HDFS-8821. Explain message "Operation category X is not supported - in state standby" (Gautam Gopalakrishnan via harsh) - - HDFS-3918. EditLogTailer shouldn't log WARN when other node - is in standby mode (todd via harsh) - - HDFS-4396. Add START_MSG/SHUTDOWN_MSG for ZKFC - (Liang Xie via harsh) - - HDFS-7875. Improve log message when wrong value configured for - dfs.datanode.failed.volumes.tolerated. - (nijel via harsh) - - HDFS-2360. Ugly stacktrace when quota exceeds. (harsh) - - HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for - DFSClient. (Zhihai Xu via Yongjun Zhang) - - HDFS-7829. Code clean up for LocatedBlock. (Takanobu Asanuma via jing9) - - HDFS-7854. Separate class DataStreamer out of DFSOutputStream. (Li Bo via - jing9) - - HDFS-7713. Implement mkdirs in the HDFS Web UI. (Ravi Prakash via wheat9) - - HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes - a lot of time if disks are busy (Rushabh S Shah via kihwal) - - HDFS-7990. IBR delete ack should not be delayed. (daryn via kihwal) - - HDFS-8004. Use KeyProviderCryptoExtension#warmUpEncryptedKeys when creating - an encryption zone. (awang via asuresh) - - HDFS-6263. Remove DRFA.MaxBackupIndex config from log4j.properties. - (Abhiraj Butala via aajisaka) - - HDFS-6408. Remove redundant definitions in log4j.properties. - (Abhiraj Butala via aajisaka) - - HDFS-7890. Improve information on Top users for metrics in - RollingWindowsManager and lower log level (J.Andreina via vinayakumarb) - - HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times. - (Vinayakumar B and Keisuke Ogiwara via Arpit Agarwal) - - HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads. - (Arpit Agarwal) - - HDFS-7671. hdfs user guide should point to the common rack awareness doc. - (Kai Sasaki via aajisaka) - - HDFS-8009. Signal congestion on the DataNode. (wheat9) - - HDFS-7978. Add LOG.isDebugEnabled() guard for some LOG.debug(..). - (Walter Su via wang) - - HDFS-7888. Change DFSOutputStream and DataStreamer for convenience of - subclassing. (Li Bo via szetszwo) - - HDFS-8049. Add @InterfaceAudience.Private annotation to hdfs client - implementation. (Takuya Fukudome via szetszwo) - - HDFS-8079. Move dfs.client.retry.* confs from DFSConfigKeys to - HdfsClientConfigKeys.Retry. (szetszwo) - - HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it - can be easily overrided. (Walter Su via vinayakumarb) - - HDFS-8080. Separate JSON related routines used by WebHdfsFileSystem to a - package local class. (wheat9) - - HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package. - (szetszwo) - - HDFS-8076. Code cleanup for DFSInputStream: use offset instead of - LocatedBlock when possible. (Zhe Zhang via wang) - - HDFS-8025. Addendum fix for HDFS-3087 Decomissioning on NN restart can - complete without blocks being replicated. (Ming Ma via wang) - - HDFS-8089. Move o.a.h.hdfs.web.resources.* to the client jars. (wheat9) - - HDFS-7979. Initialize block report IDs with a random number. (wang) - - HDFS-8101. DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS - classes at runtime. (Sean Busbey via atm) - - HDFS-8102. Separate webhdfs retry configuration keys from DFSConfigKeys. - (wheat9) - - HDFS-8100. Refactor DFSClient.Conf to a standalone class and separates - short-circuit related conf to ShortCircuitConf. (szetszwo) - - HDFS-8103. Move BlockTokenSecretManager.AccessMode into - BlockTokenIdentifier. (wheat9) - - HDFS-8084. Move dfs.client.failover.* confs from DFSConfigKeys to - HdfsClientConfigKeys.Failover and fix typos in the dfs.http.client.* - configuration keys. (szetszwo) - - HDFS-7933. fsck should also report decommissioning replicas. - (Xiaoyu Yao via cnauroth) - - HDFS-8083. Move dfs.client.write.* conf from DFSConfigKeys to - HdfsClientConfigKeys.Write. (szetszwo) - - HDFS-8117. More accurate verification in SimulatedFSDataset: replace - DEFAULT_DATABYTE with patterned data. (Zhe Zhang via wang) - - HDFS-8144. Split TestLazyPersistFiles into multiple tests. (Arpit Agarwal) - - HDFS-8082. Move dfs.client.read.*, dfs.client.short.circuit.*, - dfs.client.mmap.* and dfs.client.hedged.read.* conf from DFSConfigKeys - to HdfsClientConfigKeys. (szetszwo) - - HDFS-8165. Move GRANDFATHER_GENERATION_STAMP and GRANDFATER_INODE_ID to - hdfs-client. (wheat9) - - HDFS-8152. Refactoring of lazy persist storage cases. (Arpit Agarwal) - - HDFS-8169. Move LocatedBlocks and related classes to hdfs-client. (wheat9) - - HDFS-8133. Improve readability of deleted block check (Daryn Sharp via - Colin P. McCabe) - - HDFS-8185. Separate client related routines in HAUtil into a new class. - (wheat9) - - HDFS-8218. Move classes that used by ClientProtocol into hdfs-client. - (wheat9) - - HDFS-4448. Allow HA NN to start in secure mode with wildcard address - configured (atm via asuresh) - - HDFS-8215. Refactor NamenodeFsck#check method. (Takanobu Asanuma - via szetszwo) - - HDFS-8052. Move WebHdfsFileSystem into hadoop-hdfs-client. (wheat9) - - HDFS-8176. Record from/to snapshots in audit log for snapshot diff report. - (J. Andreina via jing9) - - HDFS-8280. Code Cleanup in DFSInputStream. (Jing Zhao via wheat9) - - HDFS-8283. DataStreamer cleanup and some minor improvement. (szetszwo via - jing9) - - HDFS-5574. Remove buffer copy in BlockReader.skip. - (Binglin Chang via aajisaka) - - HDFS-8200. Refactor FSDirStatAndListingOp. (wheat9) - - HDFS-8292. Move conditional in fmt_time from dfs-dust.js to status.html. - (Charles Lamb via wang) - - HDFS-8086. Move LeaseRenewer to the hdfs.client.impl package. (Takanobu - Asanuma via szetszwo) - - HDFS-8249. Separate HdfsConstants into the client and the server side - class. (wheat9) - - HDFS-7397. Add more detail to the documentation for the conf key - "dfs.client.read.shortcircuit.streams.cache.size" (Brahma Reddy Battula via - Colin P. McCabe) - - HDFS-8237. Move all protocol classes used by ClientProtocol to hdfs-client. - (wheat9) - - HDFS-7758. Retire FsDatasetSpi#getVolumes() and use - FsDatasetSpi#getVolumeRefs() instead (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote - NameNode (Charles Lamb via Colin P. McCabe) - - HDFS-8314. Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and - SMALL_BUFFER_SIZE to the users. (Li Lu via wheat9) - - HDFS-8207. Improper log message when blockreport interval compared with - initial delay. (Brahma Reddy Battula and Ashish Singhi via ozawa) - - HDFS-7559. Create unit test to automatically compare HDFS related classes - and hdfs-default.xml. (Ray Chiang via asuresh) - - HDFS-5640. Add snapshot methods to FileContext. (Rakesh R via cnauroth) - - HDFS-8284. Update documentation about how to use HTrace with HDFS (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-8113. Add check for null BlockCollection pointers in - BlockInfoContiguous structures (Chengbing Liu via Colin P. McCabe) - - HDFS-6757. Simplify lease manager with INodeID. (wheat9) - - HDFS-8327. Simplify quota calculations for snapshots and truncate. (wheat9) - - HDFS-8357. Consolidate parameters of INode.CleanSubtree() into a parameter - objects. (Li Lu via wheat9) - - HDFS-8255. Rename getBlockReplication to getPreferredBlockReplication. - (Contributed by Zhe Zhang) - - HDFS-6184. Capture NN's thread dump when it fails over. - (Ming Ma via aajisaka) - - HDFS-8350. Remove old webhdfs.xml and other outdated documentation stuff. - (Brahma Reddy Battula via aajisaka) - - HDFS-6888. Allow selectively audit logging ops (Chen He via vinayakumarb) - - HDFS-8397. Refactor the error handling code in DataStreamer. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-8394. Move getAdditionalBlock() and related functionalities into a - separate class. (wheat9) - - HDFS-8157. Writes to RAM DISK reserve locked memory for block files. - (Arpit Agarwal) - - HDFS-8345. Storage policy APIs must be exposed via the FileSystem - interface. (Arpit Agarwal) - - HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) - - HDFS-4383. Document the lease limits. (Arshad Mohammad via aajisaka) - - HDFS-8421. Move startFile() and related functions into FSDirWriteFileOp. - (wheat9) - - HDFS-8377. Support HTTP/2 in datanode. (Duo Zhang via wheat9) - - HDFS-8482. Rename BlockInfoContiguous to BlockInfo. (Zhe Zhang via wang) - - HDFS-8443. Document dfs.namenode.service.handler.count in hdfs-site.xml. - (J.Andreina via aajisaka) - - HDFS-8489. Subclass BlockInfo to represent contiguous blocks. - (Zhe Zhang via jing9) - - HDFS-8386. Improve synchronization of 'streamer' reference in - DFSOutputStream. (Rakesh R via wang) - - HDFS-8513. Rename BlockPlacementPolicyRackFaultTolarent to - BlockPlacementPolicyRackFaultTolerant. (wang) - - HDFS-8532. Make the visibility of DFSOutputStream#streamer member variable - to private. (Rakesh R via wang) - - HDFS-8535. Clarify that dfs usage in dfsadmin -report output includes all - block replicas. (Eddy Xu via wang) - - HDFS-8432. Introduce a minimum compatible layout version to allow downgrade - in more rolling upgrade use cases. (cnauroth) - - HDFS-8116. Cleanup uncessary if LOG.isDebugEnabled() from - RollingWindowManager. (Brahma Reddy Battula via xyao) - - HDFS-8553. Document hdfs class path options. - (Brahma Reddy Battula via cnauroth) - - HDFS-8552. Fix hdfs CLI usage message for namenode and zkfc. - (Brahma Reddy Battula via xyao) - - HDFS-8568. TestClusterId#testFormatWithEmptyClusterIdOption is failing. - (Rakesh R. via xyao) - - HDFS-8549. Abort the balancer if an upgrade is in progress. (wang) - - HDFS-8573. Move creation of restartMeta file logic from BlockReceiver to - ReplicaInPipeline. (Eddy Xu via wang) - - HDFS-7923. The DataNodes should rate-limit their full block reports by - asking the NN on heartbeat messages (cmccabe) - - HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. - (surendra singh lilhore via szetszwo) - - HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes - from HDFS-8386. (Rakesh R via szetszwo) - - HDFS-9608. Merge HDFS-7912 to trunk and branch-2 (track BlockInfo instead - of Block in UnderReplicatedBlocks and PendingReplicationBlocks). - (Zhe Zhang via wang) - - HDFS-8238. Move ClientProtocol to the hdfs-client. - (Takanobu Asanuma via wheat9) - - HDFS-8446. Separate safemode related operations in GetBlockLocations(). - (wheat9) - - HDFS-8589. Fix unused imports in BPServiceActor and BlockReportLeaseManager - (cmccabe) - - HDFS-6249. Output AclEntry in PBImageXmlWriter. - (surendra singh lilhore via aajisaka) - - HDFS-8605. Merge Refactor of DFSOutputStream from HDFS-7285 branch. - (vinayakumarb via wang) - - HDFS-8582. Support getting a list of reconfigurable config properties and - do not generate spurious reconfig warnings (Lei (Eddy) Xu via Colin P. - McCabe) - - HDFS-8192. Eviction should key off used locked memory instead of - ram disk free space. (Arpit Agarwal) - - HDFS-6564. Use slf4j instead of common-logging in hdfs-client. - (Rakesh R via wheat9) - - HDFS-8639. Add Option for NameNode HTTP port in MiniDFSClusterManager. - (Kai Sasaki via jing9) - - HDFS-8462. Implement GETXATTRS and LISTXATTRS operations for WebImageViewer. - (Jagadesh Kiran N via aajisaka) - - HDFS-8640. Make reserved RBW space visible through JMX. (kanaka kumar - avvaru via Arpit Agarwal) - - HDFS-8665. Fix replication check in DFSTestUtils#waitForReplication. (wang) - - HDFS-8546. Use try with resources in DataStorage and Storage. (wang) - - HDFS-8651. Make hadoop-hdfs-project Native code -Wall-clean (Alan Burlison - via Colin P. McCabe) - - HDFS-8653. Code cleanup for DatanodeManager, DatanodeDescriptor and - DatanodeStorageInfo. (Zhe Zhang via wang) - - HDFS-8493. Consolidate truncate() related implementation in a single class. - (Rakesh R via wheat9) - - HDFS-8635. Migrate HDFS native build to new CMake framework (Alan Burlison - via Colin P. McCabe) - - HDFS-8666. Speedup the TestMover tests. (Walter Su via jing9) - - HDFS-8703. Merge refactor of DFSInputStream from ErasureCoding branch - (vinayakumarb) - - HDFS-8709. Clarify automatic sync in FSEditLog#logEdit. (wang) - - HDFS-8711. setSpaceQuota command should print the available storage type - when input storage type is wrong. (Brahma Reddy Battula via xyao) - - HDFS-8620. Clean up the checkstyle warinings about ClientProtocol. - (Takanobu Asanuma via wheat9) - - HDFS-8712. Remove 'public' and 'abstracta modifiers in FsVolumeSpi and - FsDatasetSpi (Lei (Eddy) Xu via vinayakumarb) - - HDFS-8726. Move protobuf files that define the client-sever protocols to - hdfs-client. (wheat9) - - HDFS-8751. Remove setBlocks API from INodeFile and misc code cleanup. (Zhe - Zhang via jing9) - - HDFS-8541. Mover should exit with NO_MOVE_PROGRESS if there is no move - progress. (Surendra Singh Lilhore via szetszwo) - - HDFS-8742. Inotify: Support event for OP_TRUNCATE. - (Surendra Singh Lilhore via aajisaka) - - HDFS-8794. Improve CorruptReplicasMap#corruptReplicasMap. (yliu) - - HDFS-7483. Display information per tier on the Namenode UI. - (Benoy Antony and wheat9 via wheat9) - - HDFS-8721. Add a metric for number of encryption zones. - (Rakesh R via cnauroth) - - HDFS-8495. Consolidate append() related implementation into a single class. - (Rakesh R via wheat9) - - HDFS-8795. Improve InvalidateBlocks#node2blocks. (yliu) - - HDFS-8797. WebHdfsFileSystem creates too many connections for pread. (jing9) - - HDFS-8730. Clean up the import statements in ClientProtocol. - (Takanobu Asanuma via wheat9) - - HDFS-8735. Inotify: All events classes should implement toString() API. - (Surendra Singh Lilhore via aajisaka) - - HDFS-7858. Improve HA Namenode Failover detection on the client. (asuresh) - - HDFS-8180. AbstractFileSystem Implementation for WebHdfs. (snayak via jghoman) - - HDFS-8811. Move BlockStoragePolicy name's constants from - HdfsServerConstants.java to HdfsConstants.java (vinayakumarb) - - HDFS-8822. Add SSD storagepolicy tests in TestBlockStoragePolicy# - testDefaultPolicies (vinayakumarb) - - HDFS-8816. Improve visualization for the Datanode tab in the NN UI. (wheat9) - - HDFS-7192. DN should ignore lazyPersist hint if the writer is - not local. (Arpit Agarwal) - - HDFS-6860. BlockStateChange logs are too noisy. (Chang Li and xyao via xyao) - - HDFS-8815. DFS getStoragePolicy implementation using single RPC call - (Surendra Singh Lilhore via vinayakumarb) - - HDFS-8856. Make LeaseManager#countPath O(1). (Arpit Agarwal) - - HDFS-8772. Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. - (Walter Su via wang) - - HDFS-8818. Changes the global moveExecutor to per datanode executors and - changes MAX_SIZE_TO_MOVE to be configurable. (szetszwo) - - HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. - (Brahma Reddy Battula via jing9) - - HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang) - - HDFS-8622. Implement GETCONTENTSUMMARY operation for WebImageViewer. - (Jagadesh Kiran N via aajisaka) - - HDFS-7649. Multihoming docs should emphasize using hostnames in - configurations. (Brahma Reddy Battula via Arpit Agarwal) - - HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo) - - HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. - (Anu Engineer via xyao) - - HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging. (wang) - - HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. - (wheat9) - - HDFS-8801. Convert BlockInfoUnderConstruction as a feature. - (Jing Zhao via wheat9) - - HDFS-8880. NameNode metrics logging. (Arpit Agarwal) - - HDFS-8278. When computing max-size-to-move in Balancer, count only the - storage with remaining >= default block size. (szetszwo) - - HDFS-8435. Support CreateFlag in WebHDFS. (Jakob Homan via cdouglas) - - HDFS-8826. In Balancer, add an option to specify the source node list - so that balancer only selects blocks to move from those nodes. (szetszwo) - - HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. - (Anu Engineer via Arpit Agarwal) - - HDFS-8803. Move DfsClientConf to hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8917. Cleanup BlockInfoUnderConstruction from comments and tests. - (Zhe Zhang via jing9) - - HDFS-8884. Fail-fast check in BlockPlacementPolicyDefault#chooseTarget. - (yliu) - - HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. - (Yufei Gu via Yongjun Zhang) - - HDFS-8823. Move replication factor into individual blocks. (wheat9) - - HDFS-8934. Move ShortCircuitShm to hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8928. Improvements for BlockUnderConstructionFeature: - ReplicaUnderConstruction as a separate class and replicas as an array. - (jing9) - - HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) - - HDFS-8951. Move the shortcircuit package to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC - root in ShutdownHookManager. (Walter Su via jing9) - - HDFS-8248. Store INodeId instead of the INodeFile object in - BlockInfoContiguous. (wheat9) - - HDFS-8962. Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf. - (Mingliang Liu via wheat9) - - HDFS-8865. Improve quota initialization performance. (kihwal) - - HDFS-8938. Extract BlockToMarkCorrupt and ReplicationWork as standalone - classes from BlockManager. (Mingliang Liu via wheat9) - - HDFS-8925. Move BlockReaderLocal to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-8983. NameNode support for protected directories. (Arpit Agarwal) - - HDFS-8980. Remove unnecessary block replacement in INodeFile. (jing9) - - HDFS-8990. Move RemoteBlockReader to hdfs-client module. - (Mingliang via wheat9) - - HDFS-8946. Improve choosing datanode storage for block placement. (yliu) - - HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) - - HDFS-2070. Add more unit tests for FsShell getmerge (Daniel Templeton via - Colin P. McCabe) - - HDFS-328. Improve fs -setrep error message for invalid replication factors. - (Daniel Templeton via wang) - - HDFS-8890. Allow admin to specify which blockpools the balancer should run - on. (Chris Trezzo via mingma) - - HDFS-9002. Move o.a.h.hdfs.net/*Peer classes to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-9021. Use a yellow elephant rather than a blue one in diagram. (wang) - - HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to - hadoop-hdfs-client module. (Mingliang Liu via wheat9) - - HDFS-8984. Move replication queues related methods in FSNamesystem to - BlockManager. (wheat9) - - HDFS-9019. Adding informative message to sticky bit permission denied - exception. (xyao) - - HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-8716. Introduce a new config specifically for safe mode block count - (Chang Li via kihwal) - - HDFS-7116. Add a command to get the balancer bandwidth - (Rakesh R via vinayakumarb) - - HDFS-8974. Convert docs in xdoc format to markdown. - (Masatake Iwasaki via aajisaka) - - HDFS-6763. Initialize file system-wide quota once on transitioning to active - (kihwal) - - HDFS-9027. Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method. - (Mingliang Liu via Arpit Agarwal) - - HDFS-8996. Consolidate validateLog and scanLog in FJM#EditLogFile (Zhe - Zhang via Colin P. McCabe) - - HDFS-9010. Replace NameNode.DEFAULT_PORT with HdfsClientConfigKeys. - DFS_NAMENODE_RPC_PORT_DEFAULT config key. (Mingliang Liu via wheat9) - - HDFS-9065. Include commas on # of files, blocks, total filesystem objects - in NN Web UI. (Daniel Templeton via wheat9) - - HDFS-9008. Balancer#Parameters class could use a builder pattern. - (Chris Trezzo via mingma) - - HDFS-8953. DataNode Metrics logging (Kanaka Kumar Avvaru via vinayakumarb) - - HDFS-9082. Change the log level in WebHdfsFileSystem.initialize() from INFO - to DEBUG. (Santhosh Nayak via cnauroth) - - HDFS-7986. Allow files / directories to be deleted from the NameNode UI. - (Ravi Prakash via wheat9) - - HDFS-7995. Implement chmod in the HDFS Web UI. - (Ravi Prakash and Haohui Mai via wheat9) - - HDFS-9022. Move NameNode.getAddress() and NameNode.getUri() to - hadoop-hdfs-client. (Mingliang Liu via wheat9) - - HDFS-5802. NameNode does not check for inode type before traversing down a - path. (Xiao Chen via Yongjun Zhang) - - HDFS-9101. Remove deprecated NameNode.getUri() static helper method. - (Mingliang Liu via wheat9) - - HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to - PBHelperClient. (Mingliang Liu via wheat9) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - HDFS-9039. Separate client and server side methods of o.a.h.hdfs. - NameNodeProxies. (Mingliang Liu via wheat9) - - HDFS-8733. Keep server related definition in hdfs.proto on server side. - (Mingliang Liu via wheat9) - - HDFS-9130. Use GenericTestUtils#setLogLevel to the logging level. - (Mingliang Liu via wheat9) - - HDFS-9131 Move config keys used by hdfs-client to HdfsClientConfigKeys. - (Mingliang Liu via wheat9) - - HDFS-7529. Consolidate encryption zone related implementation into a single - class. (Rakesh R via wheat9) - - HDFS-9134. Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from - HdfsServerConstants to HdfsConstants. (Mingliang Liu via wheat9) - - HDFS-5795. RemoteBlockReader2#checkSuccess() shoud print error status. - (Xiao Chen via Yongjun Zhang) - - HDFS-9112. Improve error message for Haadmin when multiple name service IDs - are configured. (Anu Engineer via jing9) - - HDFS-9132. Pass genstamp to ReplicaAccessorBuilder. (Colin Patrick McCabe via - Lei (Eddy) Xu) - - HDFS-9133. ExternalBlockReader and ReplicaAccessor need to return -1 on read - when at EOF. (Colin Patrick McCabe via Lei (Eddy) Xu) - - HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton - via Colin P. McCabe) - - HDFS-8053. Move DFSIn/OutputStream and related classes to - hadoop-hdfs-client. (Mingliang Liu via wheat9) - - HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark - via Colin P. McCabe) - - HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. (Mingliang Liu - via wheat9) - - HDFS-9080. Update htrace version to 4.0.1 (cmccabe) - - HDFS-9148. Incorrect assert message in TestWriteToReplica#testWriteToTemporary - (Tony Wu via lei) - - HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. - (yliu) - - HDFS-9165. Move entries in META-INF/services/o.a.h.fs.FileSystem to - hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8696. Make the lower and higher watermark in the DN Netty server - configurable. (Xiaobing Zhou via wheat9) - - HDFS-8971. Remove guards when calling LOG.debug() and LOG.trace() in client - package. (Mingliang Liu via wheat9) - - HDFS-9175. Change scope of 'AccessTokenProvider.getAccessToken()' and - 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to - public (Santhosh Nayak via cnauroth) - - HDFS-9015. Refactor TestReplicationPolicy to test different block placement - policies. (Ming Ma via lei) - - HDFS-8979. Clean up checkstyle warnings in hadoop-hdfs-client module. - (Mingliang Liu via wheat9) - - HDFS-9155. OEV should treat .XML files as XML even when the file name - extension is uppercase (nijel via cmccabe) - - HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. (wheat9) - - HDFS-8164. cTime is 0 in VERSION file for newly formatted NameNode. - (Xiao Chen via Yongjun Zhang) - - HDFS-9181. Better handling of exceptions thrown during upgrade shutdown. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9110. Use Files.walkFileTree in NNUpgradeUtil#doPreUpgrade for - better efficiency. (Charlie Helin via wang) - - HDFS-8988. Use LightWeightHashSet instead of LightWeightLinkedSet in - BlockManager#excessReplicateMap. (yliu) - - HDFS-9006. Provide BlockPlacementPolicy that supports upgrade domain. - (Ming Ma via lei) - - HDFS-1172. Blocks in newly completed files are considered under-replicated - too quickly. (Masatake Iwasaki via jing9) - - HDFS-9238. Update TestFileCreation.testLeaseExpireHardLimit() to avoid using - DataNodeTestUtils.getFile(). (Tony Wu via lei) - - HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9) - - HDFS-9188. Make block corruption related tests FsDataset-agnostic. (lei) - - HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) - - HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. - (Xiao Chen via wang) - - HDFS-9251. Refactor TestWriteToReplica and TestFsDatasetImpl to avoid - explicitly creating Files in the tests code. (lei) - - HDFS-8647. Abstract BlockManager's rack policy into BlockPlacementPolicy. - (Brahma Reddy Battula via mingma) - - HDFS-9225. Fix intermittent test failure of - TestBlockManager.testBlocksAreNotUnderreplicatedInSingleRack. - (Masatake Iwasaki via wang) - - HDFS-7087. Ability to list /.reserved. (Xiao Chen via wang) - - HDFS-9264. Minor cleanup of operations on FsVolumeList#volumes. - (Walter Su via lei) - - HDFS-8808. dfs.image.transfer.bandwidthPerSec should not apply to - -bootstrapStandby (zhz) - - HDFS-4015. Safemode should count and report orphaned blocks. - (Anu Engineer via Arpit Agarwal) - - HDFS-7284. Add more debug info to - BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9284. fsck command should not print exception trace when file not - found. (Jagadesh Kiran N via wang) - - HDFS-9291. Fix TestInterDatanodeProtocol to be FsDataset-agnostic. (lei) - - HDFS-8945. Update the description about replica placement in HDFS - Architecture documentation. (Masatake Iwasaki via wang) - - HDFS-9292. Make TestFileConcorruption independent to underlying FsDataset - Implementation. (lei) - - HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs - write scenario. (Mingliang Liu via mingma) - - HDFS-9299. Give ReplicationMonitor a readable thread name (Staffan Friberg - via Colin P. McCabe) - - HDFS-9307. fuseConnect should be private to fuse_connect.c (Mingliang Liu - via Colin P. McCabe) - - HDFS-9311. Support optional offload of NameNode HA service health checks to - a separate RPC server. (cnauroth) - - HDFS-9255. Consolidate block recovery related implementation into a single - class. (Walter Su via zhz) - - HDFS-9295. Add a thorough test of the full KMS code path. - (Daniel Templeton via zhz) - - HDFS-8545. Refactor FS#getUsed() to use ContentSummary and add an API to fetch - the total file length from a specific path (J.Andreina via vinayakumarb) - - HDFS-9229. Expose size of NameNode directory as a metric. - (Surendra Singh Lilhore via zhz) - - HDFS-9339. Extend full test of KMS ACLs. (Daniel Templeton via zhz) - - HDFS-9007. Fix HDFS Balancer to honor upgrade domain policy. (Ming Ma via lei) - - HDFS-9331. Modify TestNameNodeMXBean#testNameNodeMXBeanInfo() to account for - filesystem entirely allocated for DFS use. (Tony Wu via lei) - - HDFS-9363. Add fetchReplica() to FsDatasetTestUtils to return FsDataset-agnostic - replica. (Tony Wu via lei) - - HDFS-9377. Fix findbugs warnings in FSDirSnapshotOp. - (Mingliang Liu via Yongjun Zhang) - - HDFS-9236. Missing sanity check for block size during block recovery. - (Tony Wu via Yongjun Zhang) - - HDFS-9379. Make NNThroughputBenchmark$BlockReportStats support more than 10 - datanodes. (Mingliang Liu via Arpit Agarwal) - - HDFS-9398. Make ByteArraryManager log message in one-line format. - (Mingliang Liu via szetszwo) - - HDFS-2261. AOP unit tests are not getting compiled or run. (wheat9) - - HDFS-9369. Use ctest to run tests for hadoop-hdfs-native-client. (wheat9) - - HDFS-9252. Change TestFileTruncate to use FsDatasetTestUtils to get block - file size and genstamp. (Lei (Eddy) Xu via cmccabe) - - HDFS-8056. Decommissioned dead nodes should continue to be counted as dead - after NN restart. (mingma) - - HDFS-9439. Include status of closeAck into exception message in DataNode#run. - (Xiao Chen via Yongjun Zhang) - - HDFS-9402. Switch DataNode.LOG to use slf4j. (Walter Su via wheat9) - - HDFS-9153. Pretty-format the output for DFSIO. (Kai Zheng via wheat9) - - HDFS-7988. Replace usage of ExactSizeInputStream with LimitInputStream. - (Walter Su via wheat9) - - HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess - replicas. (Xiao Chen via mingma) - - HDFS-8807. dfs.datanode.data.dir does not handle spaces between - storageType and URI correctly. (Anu Engineer via szetszwo) - - HDFS-9438. TestPipelinesFailover assumes Linux ifconfig. - (John Zhuge via Yongjun Zhang) - - HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage - type info. (xyao) - - HDFS-9269. Update the documentation and wrapper for fuse-dfs. - (Wei-Chiu Chuang via zhz) - - HDFS-9485. Make BlockManager#removeFromExcessReplicateMap accept BlockInfo - instead of Block. (Mingliang Liu via jing9) - - HDFS-9490. MiniDFSCluster should change block generation stamp via - FsDatasetTestUtils. (Tony Wu via lei) - - HDFS-9474. TestPipelinesFailover should not fail when printing debug - message. (John Zhuge via Yongjun Zhang) - - HDFS-9214. Support reconfiguring dfs.datanode.balance.max.concurrent.moves - without DN restart. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9527. The return type of FSNamesystem.getBlockCollection should be - changed to INodeFile. (szetszwo) - - HDFS-9472. concat() API does not give proper exception messages on ./reserved - relative path (Rakesh R via umamahesh) - - HDFS-9528. Cleanup namenode audit/log/exception messages. (szetszwo via umamahesh) - - HDFS-9532. Detailed exception info is lost in reportTo methods of - ErrorReportAction and ReportBadBlockAction. (Yongjun Zhang) - - HDFS-9519. Some coding improvement in SecondaryNameNode#main. - (Xiao Chen via Yongjun Zhang) - - HDFS-9514. TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; - exception being swallowed. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9535. Newly completed blocks in IBR should not be considered - under-replicated too quickly. (Mingliang Liu via jing9) - - HDFS-8894. Set SO_KEEPALIVE on DN server sockets. - (Kanaka Kumar Avvaru via wang) - - HDFS-9557. Reduce object allocation in PB conversion - (Daryn Sharp via cnauroth) - - HDFS-9552. Document types of permission checks performed for HDFS - operations. (cnauroth) - - HDFS-7779. Support changing ownership, group and replication in HDFS Web - UI. (Ravi Prakash via wheat9) - - HDFS-9498. Move code that tracks blocks with future generation stamps - to BlockManagerSafeMode. (Mingliang Liu via Arpit Agarwal) - - HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. - (Xiao Chen via zhz) - - HDFS-9630. DistCp minor refactoring and clean up. (Kai Zheng via zhz) - - HDFS-9415. Document dfs.cluster.administrators and - dfs.permissions.superusergroup. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9655. NN should start JVM pause monitor before loading fsimage. - (John Zhuge via Lei (Eddy) Xu) - - HDFS-8898. Create API and command-line argument to get quota and quota - usage without detailed content summary. (Ming Ma via kihwal) - - HDFS-9706. Log more details in debug logs in BlockReceiver's constructor. - (Xiao Chen via Yongjun Zhang) - - HDFS-9638. Improve DistCp Help and documentation. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains - INodeReference. (Xiao Chen via lei) - - HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of - -namenode. (Mingliang Liu via shv) - - HDFS-9777. Fix typos in DFSAdmin command line and documentation. - (Wei-Chiu Chuang via umamahesh) - - HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets - for DataTransferProtocol (Gary Helmling via iwasakims) - - HDFS-9644. Update encryption documentation to reflect nested EZs. (zhz) - - HDFS-9797. Log Standby exceptions thrown by RequestHedgingProxyProvider - at DEBUG Level (Inigo Goiri via asuresh) - - HDFS-9773. Remove dead code related to SimulatedFSDataset in tests. - (Brahma Reddy Battula via aajisaka) - - HDFS-8923. Add -source flag to balancer usage message. (Chris Trezzo - via szetszwo) - - HDFS-9764. DistCp doesn't print value for several arguments including - -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9084. Pagination, sorting and filtering of files/directories in the - HDFS Web UI. (Ravi Prakash via ozawa) - - HDFS-9844. Correct path creation in getTrashRoot to handle root dir. (zhz) - - HDFS-9854. Log cipher suite negotiation more verbosely - (Wei-Chiu Chuang via cnauroth) - - HDFS-9843. Document distcp options required for copying between encrypted - locations. (Xiaoyu Yao via cnauroth) - - HDFS-9831. Document webhdfs retry configuration keys introduced by - HDFS-5219/HDFS-5122. (Xiaobing Zhou via xyao) - - HDFS-9887. WebHdfs socket timeouts should be configurable. - (Austin Donnelly and Chris Douglas via xyao) - - OPTIMIZATIONS - - HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than - DFSOutputStream#writeChunk (cmccabe) - - HDFS-7433. Optimize performance of DatanodeManager's node map. - (daryn via kihwal) - - HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a - LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) - - HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via - Colin P. McCabe) - - HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) - - HDFS-8929. Add a metric to expose the timestamp of the last journal - (surendra singh lilhore via vinayakumarb) - - HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for - DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi - via Colin P. McCabe) - - HDFS-9139. Enable parallel JUnit tests for HDFS Pre-commit - (Chris Nauroth and Vinayakumar B via vinayakumarb) - - HDFS-9145. Tracking methods that hold FSNamesytemLock for too long. - (Mingliang Liu via wheat9) - - HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead - of hdfs. (Mingliang Liu via wheat9) - - HDFS-9253. Refactor tests of libhdfs into a directory. (wheat9) - - HDFS-9280. Document NFS gateway export point parameter. (Xiao Chen via zhz) - - HDFS-9297. Update TestBlockMissingException to use corruptBlockOnDataNodesByDeletingBlockFile(). - (Tony Wu via lei) - - HDFS-9168. Move client side unit test to hadoop-hdfs-client. (wheat9) - - HDFS-9312. Fix TestReplication to be FsDataset-agnostic. (lei) - - HDFS-9308. Add truncateMeta() and deleteMeta() to MiniDFSCluster. (Tony Wu via lei) - - HDFS-9372. Remove dead code in DataStorage.recoverTransitionRead. - (Duo Zhang via wheat9) - - HDFS-9282. Make data directory count and storage raw capacity related tests - FsDataset-agnostic. (Tony Wu via lei) - - HDFS-9318. considerLoad factor can be improved. (Kuhu Shukla via kihwal) - - HDFS-8335. FSNamesystem should construct FSPermissionChecker only if - permission is enabled. (Gabor Liptak via wheat9) - - HDFS-9436. Make NNThroughputBenchmark$BlockReportStats run with 10 - datanodes by default. (Mingliang Liu via shv) - - HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE - blocks. (szetszwo) - - HDFS-9715. Check storage ID uniqueness on datanode startup - (Lei (Eddy) Xu via vinayakumarb) - - HDFS-9726. Refactor IBR code to a new class. (szetszwo) - - HDFS-9686. Remove useless boxing/unboxing code. - (Kousuke Saruta via aajisaka) - - HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. - (Lin Yiqun via aajisaka) - - HDFS-9425. Expose number of blocks per volume as a metric - (Brahma Reddy Battula via vinayakumarb) - - HDFS-9710. DN can be configured to send block receipt IBRs in batches. - (szetszwo) - - BUG FIXES - - HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. - (Gautam Gopalakrishnan via harsh) - - HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown() - (Rakesh R via vinayakumarb) - - HDFS-7867. Update action param from "start" to "prepare" in rolling upgrade - javadoc (J.Andreina via vinayakumarb) - - HDFS-3325. When configuring "dfs.namenode.safemode.threshold-pct" to a value - greater or equal to 1 there is mismatch in the UI report - (J.Andreina via vinayakumarb) - - HDFS-8002. Website refers to /trash directory. (Brahma Reddy Battula via - aajisaka) - - HDFS-7261. storageMap is accessed without synchronization in - DatanodeDescriptor#updateHeartbeatState() (Brahma Reddy Battula via Colin - P. McCabe) - - HDFS-7997. The first non-existing xattr should also throw IOException. - (zhouyingchao via yliu) - - HDFS-7922. ShortCircuitCache#close is not releasing - ScheduledThreadPoolExecutors (Rakesh R via Colin P. McCabe) - - HDFS-5215. dfs.datanode.du.reserved is not considered while computing - available space ( Brahma Reddy Battula via Yongjun Zhang) - - HDFS-7725. Incorrect "nodes in service" metrics caused all writes to fail. - (Ming Ma via wang) - - HDFS-8096. DatanodeMetrics#blocksReplicated will get incremented early and - even for failed transfers (vinayakumarb) - - HDFS-7939. Two fsimage_rollback_* files are created which are not deleted - after rollback. (J.Andreina via vinayakumarb) - - HDFS-8111. NPE thrown when invalid FSImage filename given for - 'hdfs oiv_legacy' cmd ( surendra singh lilhore via vinayakumarb ) - - HDFS-7701. Support reporting per storage type quota and usage - with hadoop/hdfs shell. (Peter Shi via Arpit Agarwal) - - HDFS-6666. Abort NameNode and DataNode startup if security is enabled but - block access token is not enabled. (Vijay Bhat via cnauroth) - - HDFS-8055. NullPointerException when topology script is missing. - (Anu Engineer via cnauroth) - - HDFS-8142. DistributedFileSystem encryption zone commands should resolve - relative paths. (Rakesh R via wang) - - HDFS-7863. Missing description of some methods and parameters in javadoc of - FSDirDeleteOp. (Brahma Reddy Battula via ozawa) - - HDFS-8043. NPE in MiniDFSCluster teardown. (Brahma Reddy Battula via ozawa) - - HDFS-8173. NPE thrown at DataNode shutdown when HTTP server was not able to - create (surendra singh lilhore via vinayakumarb) - - HDFS-7993. Provide each Replica details in fsck (J.Andreina via vinayakumarb) - - HDFS-8217. During block recovery for truncate Log new Block Id in case of - copy-on-truncate is true. (vinayakumarb) - - HDFS-8231. StackTrace displayed at client while QuotaByStorageType exceeds - (J.Andreina and Xiaoyu Yao via vinayakumarb) - - HDFS-8191. Fix byte to integer casting in SimulatedFSDataset#simulatedByte. - (Zhe Zhang via wang) - - HDFS-8211. DataNode UUID is always null in the JMX counter. (Anu Engineer - via Arpit Agarwal) - - HDFS-8247. TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing. - (Xiaoyu Yao via cnauroth) - - HDFS-8206. Fix the typos in hadoop-hdfs-httpfs. (Brahma Reddy Battula via xyao) - - HDFS-8205. CommandFormat#parse() should not parse option as - value of option. (Peter Shi and Xiaoyu Yao via Arpit Agarwal) - - HDFS-8232. Missing datanode counters when using Metrics2 sink interface. - (Anu Engineer via cnauroth) - - HDFS-8214. Secondary NN Web UI shows wrong date for Last Checkpoint. (clamb via wang) - - HDFS-8300. Fix unit test failures and findbugs warning caused by HDFS-8283. - (jing9) - - HDFS-8276. LazyPersistFileScrubber should be disabled if scrubber interval - configured zero. (Surendra Singh Lilhore via Arpit Agarwal) - - HDFS-8229. LAZY_PERSIST file gets deleted after NameNode restart. - (Surendra Singh Lilhore via Arpit Agarwal) - - HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. - (xyao) - - HDFS-8290. WebHDFS calls before namesystem initialization can cause - NullPointerException. (cnauroth) - - HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows. - (Kiran Kumar M R via Xiaoyu Yao) - - HDFS-2484. checkLease should throw FileNotFoundException when file does - not exist. (Rakesh R via shv) - - HDFS-7833. DataNode reconfiguration does not recalculate valid volumes - required, based on configured failed volumes tolerated. - (Lei (Eddy) Xu via cnauroth) - - HDFS-8325. Misspelling of threshold in log4j.properties for tests. - (Brahma Reddy Battula via aajisaka) - - HDFS-8321. CacheDirectives and CachePool operations should throw - RetriableException in safemode. (wheat9) - - HDFS-8037. CheckAccess in WebHDFS silently accepts malformed FsActions - parameters. (wheat9) - - HDFS-8257. Namenode rollingUpgrade option is incorrect in document - (J.Andreina via vinayakumarb) - - HDFS-8067. haadmin prints out stale help messages (Ajith S via vinayakumarb) - - HDFS-8174. Update replication count to live rep count in fsck report. (J.Andreina) - - HDFS-6291. FSImage may be left unclosed in BootstrapStandby#doRun() - (Sanghyun Yun via vinayakumarb) - - HDFS-7998. HDFS Federation : Command mentioned to add a NN to existing - federated cluster is wrong (Ajith S via vinayakumarb) - - HDFS-8222. Remove usage of "dfsadmin -upgradeProgress" from document which - is no longer supported. (J.Andreina via aajisaka) - - HDFS-8108. Fsck should provide the info on mandatory option to be used along with "-blocks , - -locations and -racks" (J.Andreina via umamahesh) - - HDFS-8187. Remove usage of "-setStoragePolicy" and "-getStoragePolicy" using - dfsadmin cmd (as it is not been supported) (J.Andreina via vinayakumarb) - - HDFS-8175. Provide information on snapshotDiff for supporting the comparison - between snapshot and current status (J.Andreina via vinayakumarb) - - HDFS-8209. Support different number of datanode directories in MiniDFSCluster. - (surendra singh lilhore via vinayakumarb) - - HDFS-6576. Datanode log is generating at root directory in security mode - (surendra singh lilhore via vinayakumarb) - - HDFS-3384. DataStreamer thread should be closed immediatly when failed to - setup a PipelineForAppendOrRecovery (Uma Maheswara Rao G via vinayakumarb) - - HDFS-6285. tidy an error log inside BlockReceiver. (Liang Xie via umamahesh) - - HDFS-8346. libwebhdfs build fails during link due to unresolved external - symbols. (Chris Nauroth via wheat9) - - HDFS-8274. NFS configuration nfs.dump.dir not working. (Ajith S via - Arpit Agarwal) - - HDFS-8340. Fix NFS documentation of nfs.wtmax. (Ajith S via Arpit Agarwal) - - HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. - (Esteban Gutierrez via Yongjun Zhang) - - HDFS-8326. Documentation about when checkpoints are run is out of date. - (Misty Stanley-Jones via xyao) - - HDFS-8097. TestFileTruncate is failing intermittently. (Rakesh R via - Arpit Agarwal) - - HDFS-8351. Remove namenode -finalize option from document. (aajisaka) - - HDFS-8362. Java Compilation Error in TestHdfsConfigFields.java - (Arshad Mohammad via vinayakumarb) - - HDFS-8358. TestTraceAdmin fails (Masatake Iwasaki via kihwal) - - HDFS-8380. Always call addStoredBlock on blocks which have been shifted - from one storage to another (cmccabe) - - HDFS-8243. Files written by TestHostsFiles and TestNameNodeMXBean are - causing Release Audit Warnings. (Ruth Wisniewski via Arpit Agarwal) - - HDFS-7728. Avoid updating quota usage while loading edits. - (Jing Zhao via wheat9) - - HDFS-8150. Make getFileChecksum fail for blocks under construction - (J.Andreina via vinayakumarb) - - HDFS-8371. Fix test failure in TestHdfsConfigFields for spanreceiver - properties. (Ray Chiang via aajisaka) - - HDFS-8403. Eliminate retries in TestFileCreation - #testOverwriteOpenForWrite. (Arpit Agarwal via wheat9) - - HDFS-6348. SecondaryNameNode not terminating properly on runtime exceptions - (Rakesh R via vinayakumarb) - - HDFS-8454. Remove unnecessary throttling in TestDatanodeDeath. - (Arpit Agarwal) - - HDFS-8268. Port conflict log for data node server is not sufficient - (Mohammad Shahid Khan via vinayakumarb) - - HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-7401. Add block info to DFSInputStream' WARN message when it adds - node