Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8295B200BAD for ; Tue, 11 Oct 2016 06:05:36 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8133F160AE1; Tue, 11 Oct 2016 04:05:36 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4EB28160AEB for ; Tue, 11 Oct 2016 06:05:33 +0200 (CEST) Received: (qmail 4689 invoked by uid 500); 11 Oct 2016 04:05:32 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 4613 invoked by uid 99); 11 Oct 2016 04:05:32 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2016 04:05:31 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id CC547DFB78; Tue, 11 Oct 2016 04:05:31 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sjlee@apache.org To: common-commits@hadoop.apache.org Message-Id: <08f4e6d427ac4644831c44413cf5614c@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: Updated the release notes for 2.6.5 post-release (the web site will use the correct release notes). Date: Tue, 11 Oct 2016 04:05:31 +0000 (UTC) archived-at: Tue, 11 Oct 2016 04:05:36 -0000 Repository: hadoop Updated Branches: refs/heads/branch-2.6.5 e8c9fe0b4 -> 09ba74e35 Updated the release notes for 2.6.5 post-release (the web site will use the correct release notes). Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/09ba74e3 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/09ba74e3 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/09ba74e3 Branch: refs/heads/branch-2.6.5 Commit: 09ba74e3502fb9f42d20498f8c909bff0c782327 Parents: e8c9fe0 Author: Sangjin Lee Authored: Mon Oct 10 21:05:16 2016 -0700 Committer: Sangjin Lee Committed: Mon Oct 10 21:05:16 2016 -0700 ---------------------------------------------------------------------- .../src/main/docs/releasenotes.html | 820 ++++++------------- 1 file changed, 258 insertions(+), 562 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/09ba74e3/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html b/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html index 29da383..f23d12e 100644 --- a/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html +++ b/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html @@ -1,5 +1,5 @@ -Hadoop 2.6.1 Release Notes +Hadoop 2.6.5 Release Notes -

Hadoop 2.6.1 Release Notes

Hadoop 2.6.5 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements. -

Changes since Hadoop 2.6.0

Changes since Hadoop 2.6.4

YARN-4047. - Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
- ClientRMService getApplications has high scheduler lock contention
-
YARN-3999. - Major bug reported by Jian He and fixed by Jian He
- RM hangs on draining events
-
YARN-3990. - Critical bug reported by Rohith Sharma K S and fixed by Bibin A Chundatt (resourcemanager)
- AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected
-
YARN-3978. - Major improvement reported by Eric Payne and fixed by Eric Payne (timelineserver , yarn)
- Configurably turn off the saving of container info in Generic AHS
-
YARN-3850. - Blocker bug reported by Varun Saxena and fixed by Varun Saxena (log-aggregation , nodemanager)
- NM fails to read files from full disks which can lead to container logs being lost and other issues
-
YARN-3832. - Critical bug reported by Ranga Swamy and fixed by Brahma Reddy Battula (nodemanager)
- Resource Localization fails on a cluster due to existing cache directories
-
YARN-3733. - Blocker bug reported by Bibin A Chundatt and fixed by Rohith Sharma K S (resourcemanager)
- Fix DominantRC#compare() does not work as expected if cluster resource is empty
-
YARN-3725. - Blocker bug reported by Zhijie Shen and fixed by Zhijie Shen (resourcemanager , timelineserver)
- App submission via REST API is broken in secure mode due to Timeline DT service address is empty
-
YARN-3700. - Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager , webapp , yarn)
- ATS Web Performance issue at load time when large number of jobs
-
YARN-3585. - Critical bug reported by Peng Zhang and fixed by Rohith Sharma K S
- NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled
-
YARN-3526. - Major bug reported by Weiwei Yang and fixed by Weiwei Yang (resourcemanager , webapp)
- ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster
-
YARN-3493. - Critical bug reported by Sumana Sathish and fixed by Jian He (yarn)
- RM fails to come up with error "Failed to load/recover state" when mem settings are changed
-
YARN-3487. - Critical sub-task reported by Jason Lowe and fixed by Jason Lowe (capacityscheduler)
- CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
-
YARN-3464. - Critical bug reported by zhihai xu and fixed by zhihai xu (nodemanager)
- Race condition in LocalizerRunner kills localizer before localizing all resources
-
YARN-3393. - Critical bug reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
- Getting application(s) goes wrong when app finishes before starting the attempt
-
YARN-3369. - Blocker bug reported by Giovanni Matteo Fumarola and fixed by Brahma Reddy Battula (resourcemanager)
- Missing NullPointer check in AppSchedulingInfo causes RM to die
-
YARN-3287. - Major bug reported by Jonathan Eagles and fixed by Daryn Sharp
- TimelineClient kerberos authentication failure uses wrong login context.
-
YARN-3267. - Major bug reported by Prakash Ramachandran and fixed by Chang Li
- Timelineserver applies the ACL rules after applying the limit on the number of records
-
YARN-3251. - Blocker bug reported by Jason Lowe and fixed by Craig Welch
- Fix CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)
-
YARN-3249. - Minor improvement reported by Ryu Kobayashi and fixed by Ryu Kobayashi (resourcemanager)
- Add a "kill application" button to Resource Manager's Web UI
-
YARN-3242. - Critical bug reported by zhihai xu and fixed by zhihai xu (resourcemanager)
- Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client
-
YARN-3239. - Major bug reported by Hitesh Shah and fixed by Jian He
- WebAppProxy does not support a final tracking url which has query fragments and params
-
YARN-3238. - Blocker bug reported by Jason Lowe and fixed by Jason Lowe
- Connection timeouts to nodemanagers are retried at multiple levels
-
YARN-3231. - Critical bug reported by Siqi Li and fixed by Siqi Li
- FairScheduler: Changing queueMaxRunningApps interferes with pending jobs
-
YARN-3230. - Major improvement reported by Jian He and fixed by Jian He
- Clarify application states on the web UI
-
YARN-3227. - Critical bug reported by Jonathan Eagles and fixed by Zhijie Shen
- Timeline renew delegation token fails when RM user's TGT is expired
-
YARN-3222. - Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
- RMNodeImpl#ReconnectNodeTransition should send scheduler events in sequential order
-
YARN-3207. - Major bug reported by Prakash Ramachandran and fixed by Zhijie Shen (timelineserver)
- secondary filter matches entites which do not have the key being filtered for.
-
YARN-3124. - Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
- Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
-
YARN-3103. - Blocker bug reported by Jason Lowe and fixed by Jason Lowe (client)
- AMRMClientImpl does not update AMRM token properly
-
YARN-3099. - Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
- Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label.
-
YARN-3098. - Major sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler)
- Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
-
YARN-3094. - Major bug reported by Jun Gong and fixed by Jun Gong (resourcemanager)
- reset timer for liveness monitors after RM recovery
-
YARN-3092. - Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
- Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler
-
YARN-3055. - Blocker bug reported by Yi Liu and fixed by Daryn Sharp (security)
- The token is not renewed properly if it's shared by jobs (oozie) in DelegationTokenRenewer
-
YARN-3024. - Major bug reported by Chengbing Liu and fixed by Chengbing Liu (nodemanager)
- LocalizerRunner should give DIE action when all resources are localized
-
YARN-3011. - Major sub-task reported by Wang Hao and fixed by Varun Saxena (nodemanager)
- NM dies because of the failure of resource localization
-
YARN-2997. - Major bug reported by Chengbing Liu and fixed by Chengbing Liu (nodemanager)
- NM keeps sending already-sent completed containers to RM until containers are removed from context
-
YARN-2992. - Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
- ZKRMStateStore crashes due to session expiry
-
YARN-2978. - Critical bug reported by Jason Tufo and fixed by Varun Saxena
- ResourceManager crashes with NPE while getting queue info
-
YARN-2964. - Blocker bug reported by Daryn Sharp and fixed by Jian He (resourcemanager)
- RM prematurely cancels tokens for jobs that submit jobs (oozie)
-
YARN-2952. - Major bug reported by Jian He and fixed by Rohith Sharma K S
- Incorrect version check in RMStateStore
-
YARN-2922. - Major bug reported by Jason Tufo and fixed by Rohith Sharma K S (capacityscheduler , resourcemanager , scheduler)
- ConcurrentModificationException in CapacityScheduler's LeafQueue
-
YARN-2920. - Major sub-task reported by Wangda Tan and fixed by Wangda Tan
- CapacityScheduler should be notified when labels on nodes changed
+
YARN-5483. + Major improvement reported by sandflee and fixed by sandflee
+ Optimize RMAppAttempt#pullJustFinishedContainers
YARN-2918. - Major sub-task reported by Rohith Sharma K S and fixed by Wangda Tan (resourcemanager)
- Don't fail RM if queue's configured labels are not existed in cluster-node-labels
+
YARN-5462. + Major bug reported by Eric Badger and fixed by Eric Badger
+ TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
YARN-2917. - Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
- Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
+
YARN-5353. + Critical bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
+ ResourceManager can leak delegation tokens when they are shared across apps
YARN-2910. - Major bug reported by Wilfred Spiegelenburg and fixed by Wilfred Spiegelenburg (fairscheduler)
- FSLeafQueue can throw ConcurrentModificationException
-
YARN-2906. - Major bug reported by Jason Lowe and fixed by Jason Lowe (capacityscheduler)
- CapacitySchedulerPage shows HTML tags for a queue's Active Users
-
YARN-2905. - Blocker bug reported by Jason Lowe and fixed by Varun Saxena
- AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted
-
YARN-2900. - Major sub-task reported by Jonathan Eagles and fixed by Mit Desai (timelineserver)
- Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
-
YARN-2894. +
YARN-5262. Major bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
- When ACL's are enabled, if RM switches then application can not be viewed from web.
-
YARN-2890. - Major bug reported by Mit Desai and fixed by Mit Desai
- MiniYarnCluster should turn on timeline service if configured to do so
-
YARN-2874. - Blocker bug reported by Naganarasimha G R and fixed by Naganarasimha G R (resourcemanager)
- Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
-
YARN-2865. - Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
- Application recovery continuously fails with "Application with id already present. Cannot duplicate"
-
YARN-2856. - Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
- Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTED
-
YARN-2816. - Major bug reported by zhihai xu and fixed by zhihai xu (nodemanager)
- NM fail to start with NPE during container recovery
-
YARN-2766. - Major sub-task reported by Robert Kanter and fixed by Robert Kanter (timelineserver)
- ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers
-
YARN-2694. - Major sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler , resourcemanager)
- Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY
+ Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat
YARN-2637. - Critical bug reported by Wangda Tan and fixed by Craig Welch (resourcemanager)
- maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.
+
YARN-5206. + Minor bug reported by Steve Loughran and fixed by Steve Loughran (client , security)
+ RegistrySecurity includes id:pass in exception text if considered invalid
YARN-2414. - Major bug reported by Zhijie Shen and fixed by Wangda Tan (webapp)
- RM web UI: app page will crash if app is failed before any attempt has been created
-
YARN-2340. - Critical bug reported by Nishan Shetty and fixed by Rohith Sharma K S (resourcemanager , scheduler)
- NPE thrown when RM restart after queue is STOPPED. There after RM can not recovery application's and remain in standby
-
YARN-2301. - Major improvement reported by Jian He and fixed by Naganarasimha G R
- Improve yarn container command
-
YARN-2246. - Major bug reported by Devaraj K and fixed by Devaraj K (webapp)
- Job History Link in RM UI is redirecting to the URL which contains Job Id twice
-
YARN-1984. - Major sub-task reported by Jason Lowe and fixed by Varun Saxena
- LeveldbTimelineStore does not handle db exceptions properly
-
YARN-1809. - Major sub-task reported by Zhijie Shen and fixed by Xuan Gong
- Synchronize RM and Generic History Service Web-UIs
-
MAPREDUCE-6361. - Critical bug reported by Junping Du and fixed by Junping Du
- NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host
-
MAPREDUCE-6324. - Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
- Uber jobs fail to update AMRM token when it rolls over
-
MAPREDUCE-6303. - Blocker bug reported by Jason Lowe and fixed by Jason Lowe
- Read timeout when retrying a fetch error can be fatal to a reducer
-
MAPREDUCE-6300. - Minor bug reported by Siqi Li and fixed by Siqi Li
- Task list sort by task id broken
+
YARN-5197. + Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
+ RM leaks containers if running container disappears from node update
MAPREDUCE-6267. - Minor improvement reported by Chris Trezzo and fixed by Chris Trezzo
- Refactor JobSubmitter#copyAndConfigureFiles into it's own class
+
YARN-5009. + Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
+ NMLeveldbStateStoreService database can grow substantially leading to longer recovery times
MAPREDUCE-6238. - Critical bug reported by zhihai xu and fixed by zhihai xu (mrv2)
- MR2 can't run local jobs with -libjars command options which is a regression from MR1
+
YARN-4794. + Critical bug reported by Sumana Sathish and fixed by Jian He
+ Deadlock in NMClientImpl
MAPREDUCE-6237. - Major bug reported by Kannan Rajah and fixed by Kannan Rajah (mrv2)
- Multiple mappers with DBInputFormat don't work because of reusing conections
+
YARN-4785. + Major bug reported by Jayesh and fixed by Varun Vasudev (webapp)
+ inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API - cluster/scheduler
MAPREDUCE-6230. - Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
- MR AM does not survive RM restart if RM activated a new AMRM secret key
+
YARN-4773. + Minor bug reported by Jason Lowe and fixed by Jun Gong (nodemanager)
+ Log aggregation performs extraneous filesystem operations when rolling log aggregation is disabled
MAPREDUCE-6166. - Major bug reported by Eric Payne and fixed by Eric Payne (mrv2)
- Reducers do not validate checksum of map outputs when fetching directly to disk
+
YARN-4761. + Major bug reported by Sangjin Lee and fixed by Sangjin Lee (fairscheduler)
+ NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler
MAPREDUCE-5649. - Major bug reported by stanley shi and fixed by Gera Shegalov (mrv2)
- Reduce cannot use more than 2G memory for the final merge
+
YARN-4722. + Major bug reported by Jason Lowe and fixed by Jason Lowe
+ AsyncDispatcher logs redundant event queue sizes
HDFS-8863. +
YARN-4690. + Major improvement reported by Ming Ma and fixed by Ming Ma
+ Skip object allocation in FSAppAttempt#getResourceUsage when possible
+
YARN-4573. + Major sub-task reported by Takashi Ohnishi and fixed by Takashi Ohnishi (resourcemanager , test)
+ TestRMAppTransitions.testAppRunningKill and testAppKilledKilled fail on trunk
+
YARN-4459. + Major bug reported by Jun Gong and fixed by Jun Gong (nodemanager)
+ container-executor should only kill process groups
+
YARN-4393. + Major sub-task reported by Varun Saxena and fixed by Varun Saxena (test)
+ TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently
+
YARN-2046. + Major bug reported by Jason Lowe and fixed by Ming Ma (nodemanager)
+ Out of band heartbeats are sent only on container kill and possibly too early
+
MAPREDUCE-6689. + Blocker bug reported by Wangda Tan and fixed by Wangda Tan
+ MapReduce job can infinitely increase number of reducer resource requests
+
MAPREDUCE-6656. + Blocker bug reported by J.Andreina and fixed by J.Andreina
+ [NNBench] OP_DELETE operation isn't working after MAPREDUCE-6363
+
MAPREDUCE-6637. + Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (test)
+ Testcase Failure : TestFileInputFormat.testSplitLocationInfo
+
MAPREDUCE-6635. + Critical bug reported by Sergey Shelukhin and fixed by Junping Du
+ Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException
+
MAPREDUCE-6622. + Critical improvement reported by Ray Chiang and fixed by Ray Chiang (jobhistoryserver)
+ Add capability to set JHS job cache to a task-based limit
+
Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property: +1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB. +2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.
MAPREDUCE-6558. + Major bug reported by Wilfred Spiegelenburg and fixed by Wilfred Spiegelenburg (mrv1 , mrv2)
+ multibyte delimiters with compressed input files generate duplicate records
+
MAPREDUCE-6514. + Blocker bug reported by Varun Saxena and fixed by Varun Saxena (applicationmaster)
+ Job hangs as ask is not updated after ramping down of all reducers
+
MAPREDUCE-6413. + Major bug reported by Jason Lowe and fixed by zhihai xu (test)
+ TestLocalJobSubmission is failing with unknown host
+
MAPREDUCE-6302. + Critical bug reported by mai shurong and fixed by Karthik Kambatla
+ Preempt reducers after a configurable timeout irrespective of headroom
+
MAPREDUCE-6191. + Minor test reported by sam liu and fixed by sam liu (test)
+ TestJavaSerialization fails with getting incorrect MR job result
+
MAPREDUCE-5817. + Major bug reported by Sangjin Lee and fixed by Sangjin Lee (applicationmaster)
+ Mappers get rescheduled on node transition even after all reducers are completed
+
HDFS-10870. + Trivial bug reported by John Zhuge and fixed by John Zhuge (documentation)
+ Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
+
HDFS-10763. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
- The remaining space check in BlockPlacementPolicyDefault is flawed
-
HDFS-8846. - Major bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode)
- Add a unit test for INotify functionality across a layout version upgrade
-
HDFS-8486. - Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (datanode)
- DN startup may cause severe data loss
-
Public service notice: -- Every restart of a 2.6.x or 2.7.0 DN incurs a risk of unwanted block deletion. -- Apply this patch if you are running a pre-2.7.1 release.
HDFS-8480. - Critical bug reported by Zhe Zhang and fixed by Zhe Zhang
- Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them
-
HDFS-8431. - Critical bug reported by Sumana Sathish and fixed by Anu Engineer (HDFS)
- hdfs crypto class not found in Windows
-
HDFS-8404. - Major bug reported by Nathan Roberts and fixed by Nathan Roberts (namenode)
- Pending block replication can get stuck using older genstamp
-
HDFS-8384. - Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Jing Zhao (namenode)
- Allow NN to startup if there are files having a lease but are not under construction
-
HDFS-8270. - Major bug reported by Andrey Stepachev and fixed by J.Andreina (hdfs-client)
- create() always retried with hardcoded timeout when file already exists with open lease
-
Proxy level retries will not be done on AlreadyBeingCreatedExeption for create() op.
HDFS-8245. - Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah
- Standby namenode doesn't process DELETED_BLOCK if the add block request is in edit log.
-
HDFS-8219. - Major bug reported by Peter Shi and fixed by Surendra Singh Lilhore
- setStoragePolicy with folder behavior is different after cluster restart
-
HDFS-8127. - Blocker bug reported by Jing Zhao and fixed by Jing Zhao (ha)
- NameNode Failover during HA upgrade can cause DataNode to finalize upgrade
-
HDFS-8072. - Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
- Reserved RBW space is not released if client terminates while writing block
-
HDFS-8070. - Blocker bug reported by Gopal V and fixed by Colin Patrick McCabe (caching)
- Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode
-
HDFS-8046. - Major bug reported by Kihwal Lee and fixed by Kihwal Lee
- Allow better control of getContentSummary
-
HDFS-7999. - Major bug reported by zhouyingchao and fixed by zhouyingchao
- FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time
-
HDFS-7980. - Major bug reported by Hui Zheng and fixed by Walter Su
- Incremental BlockReport will dramatically slow down the startup of a namenode
-
HDFS-7960. - Critical bug reported by Lei (Eddy) Xu and fixed by Colin Patrick McCabe
- The full block report should prune zombie storages even if they're not empty
-
HDFS-7930. - Blocker bug reported by Konstantin Shvachko and fixed by Yi Liu (namenode)
- commitBlockSynchronization() does not remove locations
-
HDFS-7929. - Major bug reported by Zhe Zhang and fixed by Zhe Zhang
- inotify unable fetch pre-upgrade edit log segments once upgrade starts
-
HDFS-7915. - Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
- The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error
-
HDFS-7894. - Critical bug reported by Kihwal Lee and fixed by Brahma Reddy Battula
- Rolling upgrade readiness is not updated in jmx until query command is issued.
-
HDFS-7885. - Critical bug reported by vitthal (Suhas) Gogate and fixed by Tsz Wo Nicholas Sze (datanode)
- Datanode should not trust the generation stamp provided by client
-
HDFS-7884. - Blocker bug reported by Tsz Wo Nicholas Sze and fixed by Brahma Reddy Battula (datanode)
- NullPointerException in BlockSender
-
HDFS-7871. - Critical bug reported by Jing Zhao and fixed by Jing Zhao
- NameNodeEditLogRoller can keep printing "Swallowing exception" message
-
HDFS-7830. - Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
- DataNode does not release the volume lock when adding a volume fails.
+ Open files can leak permanently due to inconsistent lease update
HDFS-7788. - Blocker bug reported by Kihwal Lee and fixed by Rushabh S Shah
- Post-2.6 namenode may not start up with an image containing inodes created with an old release.
+
HDFS-10653. + Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
+ Optimize conversion from path string to components
HDFS-7763. - Major bug reported by Liang Xie and fixed by Liang Xie (ha)
- fix zkfc hung issue due to not catching exception in a corner case
+
HDFS-10544. + Major bug reported by Zhe Zhang and fixed by Zhe Zhang (balancer & mover , ha)
+ Balancer doesn't work with IPFailoverProxyProvider
HDFS-7742. - Major bug reported by Nathan Roberts and fixed by Nathan Roberts (namenode)
- favoring decommissioning node for replication can cause a block to stay underreplicated for long periods
+
HDFS-10458. + Major bug reported by Zhe Zhang and fixed by Zhe Zhang (encryption , namenode)
+ getFileEncryptionInfo should return quickly for non-encrypted cluster
HDFS-7733. - Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (nfs)
- NFS: readdir/readdirplus return null directory attribute on failure
+
HDFS-10377. + Major improvement reported by Konstantin Shvachko and fixed by Yiqun Lin (logging , namenode)
+ CacheReplicationMonitor shutdown log message should use INFO level.
HDFS-7714. - Major bug reported by Chris Nauroth and fixed by Vinayakumar B (datanode)
- Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.
+
HDFS-10271. + Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
+ Extra bytes are getting released from reservedSpace for append
HDFS-7707. - Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (namenode)
- Edit log corruption due to delayed block removal again
+
HDFS-10264. + Major improvement reported by Konstantin Shvachko and fixed by Xiaobing Zhou (namenode)
+ Logging improvements in FSImageFormatProtobuf.Saver
HDFS-7610. - Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
- Fix removal of dynamically added DN volumes
-
HDFS-7609. - Critical bug reported by Carrey Zhan and fixed by Ming Ma (namenode)
- Avoid retry cache collision when Standby NameNode loading edits
-
HDFS-7596. - Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
- NameNode should prune dead storages from storageMap
-
HDFS-7587. - Blocker bug reported by Kihwal Lee and fixed by Jing Zhao (namenode)
- Edit log corruption can happen if append fails with a quota violation
-
HDFS-7579. - Minor improvement reported by Charles Lamb and fixed by Charles Lamb (datanode)
- Improve log reporting during block report rpc failure
-
HDFS-7575. - Critical bug reported by Lars Francke and fixed by Arpit Agarwal
- Upgrade should generate a unique storage ID for each volume
-
HDFS-7552. - Major bug reported by Liang Xie and fixed by Liang Xie (datanode , test)
- change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
+
HDFS-10182. + Major bug reported by zhouyingchao and fixed by zhouyingchao
+ Hedged read might overwrite user's buf
HDFS-7533. - Major bug reported by Kihwal Lee and fixed by Eric Payne
- Datanode sometimes does not shutdown on receiving upgrade shutdown command
+
HDFS-10178. + Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
+ Permanent write failures can happen if pipeline recoveries occur for the first packet
HDFS-7531. - Major improvement reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
- Improve the concurrent access on FsVolumeList
+
HDFS-9752. + Critical bug reported by Kihwal Lee and fixed by Walter Su
+ Permanent write failures may happen to slow writers during datanode rolling upgrades
HDFS-7503. - Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
- Namenode restart after large deletions can cause slow processReport (due to logging)
+
HDFS-9696. + Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
+ Garbage snapshot records lingering forever
HDFS-7489. - Critical bug reported by Noah Lorang and fixed by Noah Lorang (datanode)
- Incorrect locking in FsVolumeList#checkDirs can hang datanodes
+
HDFS-9688. + Major test reported by Zhe Zhang and fixed by Zhe Zhang (encryption , test)
+ Test the effect of nested encryption zones in HDFS downgrade
HDFS-7470. - Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong (namenode)
- SecondaryNameNode need twice memory when calling reloadFromImageFile
+
HDFS-9669. + Major improvement reported by Elliott Clark and fixed by Elliott Clark
+ TcpPeerServer should respect ipc.server.listen.queue.size
HDFS-7446. - Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
- HDFS inotify should have the ability to determine what txid it has read up to
+
HDFS-9629. + Major improvement reported by Xiao Chen and fixed by Xiao Chen
+ Update the footer of Web UI to show year 2016
HDFS-7443. - Blocker bug reported by Kihwal Lee and fixed by Colin Patrick McCabe
- Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
+
HDFS-9530. + Critical bug reported by Fei Hui and fixed by Brahma Reddy Battula (datanode)
+ ReservedSpace is not cleared for abandoned Blocks
HDFS-7425. - Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)
- NameNode block deletion logging uses incorrect appender.
+
HDFS-9365. + Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
+ Balancer does not work with the HDFS-6376 HA setup
HDFS-7314. - Major improvement reported by Ming Ma and fixed by Ming Ma
- When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient
+
HDFS-9347. + Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (test)
+ Invariant assumption in TestQuorumJournalManager.shutdown() is wrong
HDFS-7278. - Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)
- Add a command that allows sysadmins to manually trigger full block reports from a DN
+
HDFS-8845. + Major bug reported by Chang Li and fixed by Chang Li
+ DiskChecker should not traverse the entire tree
HDFS-7263. - Major bug reported by Konstantin Shvachko and fixed by Tao Luo (hdfs-client)
- Snapshot read can reveal future bytes for appended files.
+
HDFS-8581. + Minor bug reported by tongshiquan and fixed by J.Andreina (namenode)
+ ContentSummary on / skips further counts on yielding lock
HDFS-7235. - Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode , namenode)
- DataNode#transferBlock should report blocks that don't exist using reportBadBlock
+
HDFS-7530. + Minor bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
+ Allow renaming of encryption zone roots
HDFS-7225. - Major bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode)
- Remove stale block invalidation work when DN re-registers with different UUID
+
HDFS-7258. + Minor bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (namenode)
+ CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level
HDFS-7213. - Critical bug reported by Daryn Sharp and fixed by Eric Payne (namenode)
- processIncrementalBlockReport performance degradation
+
HDFS-2580. + Minor improvement reported by Harsh J and fixed by Harsh J (namenode)
+ NameNode#main(...) can make use of GenericOptionsParser.
HDFS-7182. - Major improvement reported by Ming Ma and fixed by Ming Ma
- JMX metrics aren't accessible when NN is busy
+
HADOOP-13579. + Blocker bug reported by Akira Ajisaka and fixed by Tsuyoshi Ozawa
+ Fix source-level compatibility after HADOOP-11252
HDFS-7035. - Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
- Make adding a new data directory to the DataNode an atomic operation and improve error handling
+
HADOOP-13494. + Major bug reported by Sean Mackrory and fixed by Sean Mackrory (security)
+ ReconfigurableBase can log sensitive information
HDFS-7009. - Major bug reported by Ming Ma and fixed by Ming Ma (datanode)
- Active NN and standby NN have different live nodes
+
HADOOP-13434. + Major bug reported by Owen O'Malley and fixed by Owen O'Malley
+ Add quoting to Shell class
HDFS-4882. - Critical bug reported by Zesheng Wu and fixed by Ravi Prakash (hdfs-client , namenode)
- Prevent the Namenode's LeaseManager from looping forever in checkLeases
+
HADOOP-13350. + Blocker bug reported by Xiao Chen and fixed by Xiao Chen (build)
+ Additional fix to LICENSE and NOTICE
HDFS-3443. - Major bug reported by suja s and fixed by Vinayakumar B (auto-failover , ha)
- Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer
+
HADOOP-13298. + Minor improvement reported by Xiao Chen and fixed by Tsuyoshi Ozawa
+ Fix the leftover L&N files in hadoop-build-tools/src/main/resources/META-INF/
HADOOP-12280. - Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (test)
- Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded
+
HADOOP-13297. + Major bug reported by Akira Ajisaka and fixed by Sean Busbey (build)
+ Add missing dependency in setting maven-remote-resource-plugin to fix builds
HADOOP-11934. - Blocker bug reported by Mike Yoder and fixed by Larry McCay (security)
- Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop
+
HADOOP-13290. + Major improvement reported by Konstantin Shvachko and fixed by Jonathan Hung (ipc)
+ Appropriate use of generics in FairCallQueue
HADOOP-11932. - Critical bug reported by Jian He and fixed by Brahma Reddy Battula
- MetricsSinkAdapter hangs when being stopped
+
HADOOP-13255. + Major bug reported by Xiao Chen and fixed by Xiao Chen (kms)
+ KMSClientProvider should check and renew tgt when doing delegation token operations.
HADOOP-11812. - Blocker improvement reported by Gera Shegalov and fixed by Gera Shegalov (fs)
- Implement listLocatedStatus for ViewFileSystem to speed up split calculation
+
HADOOP-13192. + Critical bug reported by binde and fixed by binde (util)
+ org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly
HADOOP-11802. - Major bug reported by Eric Payne and fixed by Colin Patrick McCabe
- DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm
+
HADOOP-13189. + Major bug reported by Konstantin Shvachko and fixed by Vinitha Reddy Gankidi (ipc)
+ FairCallQueue makes callQueue larger than the configured capacity.
HADOOP-11730. - Major bug reported by Takenori Sato and fixed by Takenori Sato (fs/s3)
- Regression: s3n read failure recovery broken
+
HADOOP-13052. + Major bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ ChecksumFileSystem mishandles crc file permissions
HADOOP-11710. - Critical sub-task reported by Sean Busbey and fixed by Sean Busbey (fs)
- Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
+
HADOOP-13043. + Major bug reported by Andrew Wang and fixed by Andrew Wang
+ Add LICENSE.txt entries for bundled javascript dependencies
HADOOP-11674. - Critical bug reported by Sean Busbey and fixed by Sean Busbey (io)
- oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static
+
HADOOP-13042. + Major bug reported by Andrew Wang and fixed by Andrew Wang
+ Restore lost leveldbjni LICENSE and NOTICE changes
HADOOP-11604. - Critical bug reported by Liang Xie and fixed by Chris Nauroth (net)
- Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread.
+
HADOOP-12958. + Major bug reported by Jason Lowe and fixed by Sangjin Lee
+ PhantomReference for filesystem statistics can trigger OOM
HADOOP-11506. - Major improvement reported by Dmitriy V. Ryaboy and fixed by Gera Shegalov (conf)
- Configuration variable expansion regex expensive for long values
+
HADOOP-12893. + Blocker bug reported by Allen Wittenauer and fixed by Xiao Chen (build)
+ Verify LICENSE.txt and NOTICE.txt
HADOOP-11491. - Critical bug reported by Gera Shegalov and fixed by Brahma Reddy Battula (fs)
- HarFs incorrectly declared as requiring an authority
+
HADOOP-12810. + Major bug reported by Ryan Blue and fixed by Ryan Blue (fs , fs/s3)
+ FileSystem#listLocatedStatus causes unnecessary RPC calls
HADOOP-11482. - Major bug reported by Arun Suresh and fixed by Arun Suresh
- Use correct UGI when KMSClientProvider is called by a proxy user
+
HADOOP-12805. + Major improvement reported by Ted Yu and fixed by Ted Yu
+ Annotate CanUnbuffer with @InterfaceAudience.Public
+
Made CanBuffer interface public for use in client applications.
HADOOP-12800. + Major improvement reported by Zhe Zhang and fixed by Zhe Zhang (build , yetus)
+ Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs
HADOOP-11466. - Minor improvement reported by Suman Somasundar and fixed by Suman Somasundar (io , performance , util)
- FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower there
+
HADOOP-12789. + Minor improvement reported by Sangjin Lee and fixed by Sangjin Lee (util)
+ log classpath of ApplicationClassLoader at INFO level
HADOOP-11368. - Major bug reported by Arun Suresh and fixed by Arun Suresh (kms)
- Fix SSLFactory truststore reloader thread leak in KMSClientProvider
+
HADOOP-12773. + Major bug reported by Sangjin Lee and fixed by Sangjin Lee (util)
+ HBase classes fail to load with client/job classloader enabled
HADOOP-11350. - Major bug reported by Benoy Antony and fixed by Benoy Antony (security)
- The size of header buffer of HttpServer is too small when HTTPS is enabled
+
HADOOP-12682. + Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang
+ Fix TestKMS#testKMSRestart* failure
HADOOP-11343. - Blocker bug reported by Jerry Chen and fixed by Jerry Chen (security)
- Overflow is not properly handled in caclulating final iv for AES CTR
+
HADOOP-12589. + Major bug reported by Tsuyoshi Ozawa and fixed by Masatake Iwasaki (test)
+ Fix intermittent test failure of TestCopyPreserveFlag
HADOOP-11333. - Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong
- Fix deadlock in DomainSocketWatcher when the notification pipe is full
+
HADOOP-12559. + Major bug reported by Zhe Zhang and fixed by Zhe Zhang (security)
+ KMS connection failures should trigger TGT renewal
HADOOP-11316. - Blocker bug reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa
- "mvn package -Pdist,docs -DskipTests -Dtar" fails because of non-ascii characters
+
HADOOP-12482. + Major bug reported by Tony Wu and fixed by Tony Wu
+ Race condition in JMX cache update
HADOOP-11295. - Major bug reported by Ming Ma and fixed by Ming Ma
- RPC Server Reader thread can't shutdown if RPCCallQueue is full
+
HADOOP-12348. + Major bug reported by zhihai xu and fixed by zhihai xu (metrics)
+ MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.
HADOOP-11238. - Minor bug reported by Chris Li and fixed by Chris Li
- Update the NameNode's Group Cache in the background when possible
+
HADOOP-11361. + Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
+ Fix a race condition in MetricsSourceAdapter.updateJmxCache
HADOOP-10786. - Major bug reported by Tobi Vollebregt and fixed by Stephen Chu (security)
- Fix UGI#reloginFromKeytab on Java 8
+
HADOOP-11301. + Major improvement reported by Maysam Yabandeh and fixed by Maysam Yabandeh
+ [optionally] update jmx cache to drop old metrics
HADOOP-8151. - Major bug reported by Todd Lipcon and fixed by Matt Foley (io , native)
- Error handling in snappy decompressor throws invalid exceptions
+
HADOOP-7817. + Major bug reported by Kristofer Tomasette and fixed by Kanaka Kumar Avvaru (fs)
+ RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()
HADOOP-7139. - Major improvement reported by Stephen Rose and fixed by Kanaka Kumar Avvaru (io)
- Allow appending to existing SequenceFiles
-
Existing sequence files can be appended.

--------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-commits-help@hadoop.apache.org