hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s..@apache.org
Subject svn commit: r1207774 [2/2] - in /hadoop/common/branches/branch-0.22/common: CHANGES.txt build.xml src/docs/releasenotes.html
Date Tue, 29 Nov 2011 07:51:27 GMT
Modified: hadoop/common/branches/branch-0.22/common/src/docs/releasenotes.html
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.22/common/src/docs/releasenotes.html?rev=1207774&r1=1207773&r2=1207774&view=diff
==============================================================================
--- hadoop/common/branches/branch-0.22/common/src/docs/releasenotes.html (original)
+++ hadoop/common/branches/branch-0.22/common/src/docs/releasenotes.html Tue Nov 29 07:51:25 2011
@@ -1 +1,4790 @@
-THIS IS A PLACEHOLDER.  REAL RELEASE NOTES WILL BE ADDED TO THIS FILE IN RELEASE BRANCHES.
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+<title>Hadoop 0.22.0 Release Notes</title>
+<STYLE type="text/css">
+    H1 {font-family: sans-serif}
+    H2 {font-family: sans-serif; margin-left: 7mm}
+    TABLE {margin-left: 7mm}
+  </STYLE>
+</head>
+
+<body>
+<h1>Hadoop 0.22.0 Release Notes</h1>
+    These release notes include new developer and user-facing incompatibilities, features, and major improvements. 
+
+<a name="changes"/>
+<h2>Changes since Hadoop 0.21.1</h2>
+<ul>
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7730">HADOOP-7730</a>.
+     Major test reported by cos and fixed by cos (test)<br>
+     <b>Allow TestCLI to be run against a cluster</b><br>
+     <blockquote>Use the same CLI test to test cluster bits (see HDFS-1762 for more info)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7861">HADOOP-7861</a>.
+     Major improvement reported by shv and fixed by shv (documentation)<br>
+     <b>changes2html.pl should generate links to HADOOP, HDFS, and MAPREDUCE jiras</b><br>
+     <blockquote>changes2html.pl correctly generates links to HADOOP jiras only. This hasn't been updated since projects split.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7786">HADOOP-7786</a>.
+     Major improvement reported by eli and fixed by eli <br>
+     <b>Remove HDFS-specific configuration keys defined in FsConfig</b><br>
+     <blockquote>HADOOP-4952 added a couple HDFS-specific configuration values to common (the block size and the replication factor) that conflict with the HDFS values (eg have the wrong defaults, wrong key name), are not used by common or hdfs and should be removed. After removing these I noticed the rest of FsConfig is only used once outside a test, and isn&apos;t tagged as a public API, I think we can remove it entirely.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7783">HADOOP-7783</a>.
+     Major test reported by eli and fixed by eli (fs)<br>
+     <b>Add more symlink tests that cover intermediate links</b><br>
+     <blockquote>This covers the tests for HDFS-2514.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7697">HADOOP-7697</a>.
+     Major bug reported by shv and fixed by shv (build)<br>
+     <b>Remove dependency on different version of slf4j in avro</b><br>
+     <blockquote>Avro upgrade led to a mixture of slf4j versions. Hadoop uses slf4j 1.5.11, and avro brings in 1.6.1</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7663">HADOOP-7663</a>.
+     Major bug reported by mayank_bansal and fixed by mayank_bansal (test)<br>
+     <b>TestHDFSTrash failing on 22</b><br>
+     <blockquote>Seems to have started failing recently in many commit builds as well as the last two nightly builds of 22:
+<br>https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/51/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testTrashEmptier/
+<br>
+<br>https://issues.apache.org/jira/browse/HDFS-1967</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7646">HADOOP-7646</a>.
+     Major bug reported by jrottinghuis and fixed by jrottinghuis (io, ipc)<br>
+     <b>Make hadoop-common use same version of avro as HBase</b><br>
+     <blockquote>HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
+<br>When building HBase on top of hadoop, this should be consistent.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7577">HADOOP-7577</a>.
+     Minor bug reported by jrottinghuis and fixed by jrottinghuis (metrics)<br>
+     <b>TT does not start due to backwards compatibility wrt. EventCounter</b><br>
+     <blockquote>Between metrics1 and mertrics2 EventCounter was moved from o.a.h.log to o.a.h.metrics.jvm.
+<br>On 0.20-security a wrapper marked with @Deprecated was added back to o.a.h.log for compatibility, the same wrapper exists on trunk, but no on 0.22.
+<br>
+<br>Without it the TT will fail to start with a ClassNotFoundException.
+<br>Hive configuration also point to this class in the log4j.properties.
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7568">HADOOP-7568</a>.
+     Major bug reported by shv and fixed by zero45 (io)<br>
+     <b>SequenceFile should not print into stdout</b><br>
+     <blockquote>The following line in {{SequenceFile.Reader.initialize()}} should be removed:
+<br>System.out.println(&quot;Setting end to &quot; + end);
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7514">HADOOP-7514</a>.
+     Minor bug reported by jrottinghuis and fixed by jrottinghuis (build)<br>
+     <b>Build fails with ClassCastException when running both mvn-install and mvn-deploy targets</b><br>
+     <blockquote>Although this may not be a common use-case, the exception thrown is really confusing and does not clarify what the problem is.
+<br>The resulting error is: java.lang.ClassCastException: org.codehaus.plexus.DefaultPlexusContainer cannot be cast to org.codehaus.plexus.PlexusContainer
+<br>The error occurs because mvn-init target gets called twice.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7513">HADOOP-7513</a>.
+     Major bug reported by jrottinghuis and fixed by jrottinghuis (build)<br>
+     <b>mvn-deploy target fails</b><br>
+     <blockquote>When executing mvn-deploy target, the build fails.
+<br>hadoop-common and hadoop-common-sources deploy, but the test jar does not.
+<br>
+<br>property staging is not set and/or set to false, meaning when you try to deploy a snapshot build.
+<br>
+<br>The error reads:
+<br>Invalid reference: &apos;hadoop.core.test&apos;.
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7457">HADOOP-7457</a>.
+     Blocker improvement reported by jghoman and fixed by jghoman (documentation)<br>
+     <b>Remove out-of-date Chinese language documentation</b><br>
+     <blockquote>The Chinese language documentation haven&apos;t been updated (other than copyright years and svn moves) since their original contribution several years ago.  Worse than no docs is out-of-date, wrong docs.  We should delete them from the source tree.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7450">HADOOP-7450</a>.
+     Blocker bug reported by jnp and fixed by cos (build)<br>
+     <b>Bump jetty to 6.1.26</b><br>
+     <blockquote>Bump the jetty version, as previous version has an issue that can cause it to hang at startup.
+<br>
+<br>6.1.14 jetty is also tends to hung on heavy datanode loads.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7390">HADOOP-7390</a>.
+     Minor bug reported by tgraves and fixed by tlipcon (build)<br>
+     <b>VersionInfo not generated properly in git after unsplit</b><br>
+     <blockquote>The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
+<br>
+<br>@HadoopVersionAnnotation(version=&quot;0.22.0-SNAPSHOT&quot;, revision=&quot;Unknown&quot;, branch=&quot;Unknown&quot;,
+<br>                         user=&quot;tgraves&quot;, date=&quot;Tue Jun 14 13:39:10 UTC 2011&quot;, url=&quot;file:///home/tgraves/git/hadoop-common/common&quot;,
+<br>                         srcChecksum=&quot;0f78ea668971fe51e7ebf4f97f84eed2&quot;)
+<br>
+<br>The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7358">HADOOP-7358</a>.
+     Minor improvement reported by tlipcon and fixed by tlipcon (ipc)<br>
+     <b>Improve log levels when exceptions caught in RPC handler</b><br>
+     <blockquote>When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.
+<br>
+<br>I&apos;d like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7355">HADOOP-7355</a>.
+     Major improvement reported by stack and fixed by stack <br>
+     <b>Add audience and stability annotations to HttpServer class</b><br>
+     <blockquote>HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7351">HADOOP-7351</a>.
+     Major bug reported by stack and fixed by stack <br>
+     <b>Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461</b><br>
+     <blockquote>It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
+<br>
+<br>-  protected String getWebAppsPath() throws IOException {
+<br>-    URL url = getClass().getClassLoader().getResource(&quot;webapps&quot;);
+<br>+  private String getWebAppsPath(String appName) throws FileNotFoundException {
+<br>+    URL url = getClass().getClassLoader().getResource(&quot;webapps/&quot; + appName);
+<br>...
+<br>
+<br>HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
+<br>
+<br>This change made it into 0.21.  I&apos;d like to get a fix committed to 0.22 as well as TRUNK.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7349">HADOOP-7349</a>.
+     Major bug reported by tlipcon and fixed by tlipcon (ipc, test)<br>
+     <b>HADOOP-7121 accidentally disabled some tests</b><br>
+     <blockquote>When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7346">HADOOP-7346</a>.
+     Major improvement reported by tlipcon and fixed by tlipcon (ipc)<br>
+     <b>Send back nicer error to clients using outdated IPC version</b><br>
+     <blockquote>When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like &quot;EOFException&quot;.
+<br>
+<br>Instead, the IPC server code can speak just enough of prior IPC protocols to send back a &quot;fatal&quot; message indicating the version mismatch.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7335">HADOOP-7335</a>.
+     Minor improvement reported by tlipcon and fixed by tlipcon (build, test)<br>
+     <b>Force entropy to come from non-true random for tests</b><br>
+     <blockquote>Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
+<br>
+<br>We should turn this on for the test targets by default, so developers/hudson boxes don&apos;t have to make this change system-wide or use workarounds like rngtools.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7325">HADOOP-7325</a>.
+     Minor improvement reported by brocknoland and fixed by brocknoland (scripts)<br>
+     <b>hadoop command - do not accept class names starting with a hyphen</b><br>
+     <blockquote>
+The hadoop command does not appear to advertise allowing JVM options before the classname.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7318">HADOOP-7318</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon (io)<br>
+     <b>MD5Hash factory should reset the digester it returns</b><br>
+     <blockquote>Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it&apos;s a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7312">HADOOP-7312</a>.
+     Minor bug reported by tlipcon and fixed by qwertymaniac (conf)<br>
+     <b>core-default.xml lists configuration version as 0.21</b><br>
+     <blockquote>This key was added in HADOOP-6233, though appears unused. I suppose it&apos;s somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
+<br>
+<br>Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7302">HADOOP-7302</a>.
+     Major bug reported by asrabkin and fixed by asrabkin (documentation)<br>
+     <b>webinterface.private.actions should not be in common</b><br>
+     <blockquote>The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
+<br>
+<br>I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7300">HADOOP-7300</a>.
+     Major bug reported by tlipcon and fixed by tlipcon (conf)<br>
+     <b>Configuration methods that return collections are inconsistent about mutability</b><br>
+     <blockquote>In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
+<br>
+<br>IMO we should always return mutable collections since these methods by definition are doing copies.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7298">HADOOP-7298</a>.
+     Major test reported by tlipcon and fixed by tlipcon (test)<br>
+     <b>Add test utility for writing multi-threaded tests</b><br>
+     <blockquote>A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It&apos;s often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
+<br>
+<br>In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I&apos;d like to copy that utility into Hadoop so we can use it here as well.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7296">HADOOP-7296</a>.
+     Minor bug reported by sseth and fixed by sseth (fs)<br>
+     <b>The FsPermission(FsPermission) constructor does not use the sticky bit</b><br>
+     <blockquote>The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7291">HADOOP-7291</a>.
+     Major task reported by eli and fixed by eli <br>
+     <b>Update Hudson job not to run test-contrib</b><br>
+     <blockquote>The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn&apos;t execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  </blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7287">HADOOP-7287</a>.
+     Blocker bug reported by tlipcon and fixed by atm (conf)<br>
+     <b>Configuration deprecation mechanism doesn&apos;t work properly for GenericOptionsParser/Tools</b><br>
+     <blockquote>For example, you can&apos;t use -D options on the &quot;hadoop fs&quot; command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
+<br>- JVM starts
+<br>- GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
+<br>- DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
+<br>- Some class calls conf.get(&quot;new key&quot;) and sees the default instead of the version set on the command line</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7283">HADOOP-7283</a>.
+     Blocker task reported by tomwhite and fixed by tomwhite (build)<br>
+     <b>Include 32-bit and 64-bit native libraries in Jenkins tarball builds</b><br>
+     <blockquote>The job at https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/ is building tarballs, but they do not currently include both 32-bit and 64-bit native libraries. We should update/duplicate hadoop-nighly/hudsonBuildHadoopRelease.sh to support post-split builds.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7252">HADOOP-7252</a>.
+     Minor bug reported by ponyloky and fixed by qwertymaniac (build, conf, test)<br>
+     <b>JUnit shows up as a compile time dependency</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7245">HADOOP-7245</a>.
+     Major bug reported by tomwhite and fixed by tomwhite <br>
+     <b>FsConfig should use constants in CommonConfigurationKeys</b><br>
+     <blockquote>In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7244">HADOOP-7244</a>.
+     Blocker improvement reported by tomwhite and fixed by tomwhite (documentation)<br>
+     <b>Documentation change for updated configuration keys</b><br>
+     <blockquote>Common counterpart of HDFS-671.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7241">HADOOP-7241</a>.
+     Minor improvement reported by weiyj and fixed by weiyj (fs, test)<br>
+     <b>fix typo of command &apos;hadoop fs -help tail&apos;</b><br>
+     <blockquote>Fix the typo of command &apos;hadoop fs -help tail&apos;.
+<br>
+<br>$ hadoop fs -help tail
+<br>-tail [-f] &lt;file&gt;:  Show the last 1KB of the file. 
+<br>    The -f option shows apended data as the file grows. 
+<br>
+<br>The &quot;apended data&quot; should be &quot;appended data&quot;.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7229">HADOOP-7229</a>.
+     Major bug reported by atm and fixed by atm (security)<br>
+     <b>Absolute path to kinit in auto-renewal thread</b><br>
+     <blockquote>In the auto-renewal thread for Kerberos credentials in {{UserGroupInformation}}, the path to {{kinit}} is defaulted to {{/usr/kerberos/bin/kinit}}. This is the default path to {{kinit}} on RHEL/CentOS for MIT krb5, but not on Debian/Ubuntu (and perhaps others OSes.)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7192">HADOOP-7192</a>.
+     Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (documentation)<br>
+     <b>fs -stat docs aren&apos;t updated to reflect the format features</b><br>
+     <blockquote>The html docs of the &apos;fs -stat&apos; command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
+<br>
+<br>Like &apos;fs -help&apos;, the docs must also reflect the latest available features.
+<br>
+<br>I shall attach a doc-fix patch shortly.
+<br>
+<br>If anyone has other discrepancies to point out in the web version of the guide, please do so :)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7189">HADOOP-7189</a>.
+     Minor improvement reported by tlipcon and fixed by yuzhihong@gmail.com (security)<br>
+     <b>Add ability to enable &apos;debug&apos; property in JAAS configuration</b><br>
+     <blockquote>Occasionally users have run into weird &quot;Unable to login&quot; messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the &quot;debug&quot; option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7187">HADOOP-7187</a>.
+     Major bug reported by umamaheswararao and fixed by umamaheswararao (metrics)<br>
+     <b>Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext</b><br>
+     <blockquote>Init method is creating DatagramSocket. But this is not closed any where. 
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7184">HADOOP-7184</a>.
+     Major bug reported by tlipcon and fixed by tlipcon (documentation, filecache)<br>
+     <b>Remove deprecated local.cache.size from core-default.xml</b><br>
+     <blockquote>MAPREDUCE-2379 documents the new name of this parameter (mapreduce.tasktracker.cache.local.size) in mapred-default.xml where it belongs.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7172">HADOOP-7172</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon (io, security)<br>
+     <b>SecureIO should not check owner on non-secure clusters that have no native support</b><br>
+     <blockquote>The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to &quot;ls -ld&quot; which is very very slow. We&apos;ve seen this cause significant performance regressions on clusters that match this profile.
+<br>
+<br>Since the racy permissions check doesn&apos;t buy us any security anyway, we should just fall back to a normal &quot;open&quot; without any stat() at all, if we can&apos;t use the native support to do it efficiently.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7156">HADOOP-7156</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon <br>
+     <b>getpwuid_r is not thread-safe on RHEL6</b><br>
+     <blockquote>Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
+<br>
+<br>https://fedorahosted.org/sssd/ticket/640
+<br>
+<br>This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
+<br>
+<br>*** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
+<br>======= Backtrace: =========
+<br>/lib64/libc.so.6[0x3575675676]
+<br>/lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
+<br>/lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7154">HADOOP-7154</a>.
+     Minor improvement reported by tlipcon and fixed by tlipcon (scripts)<br>
+     <b>Should set MALLOC_ARENA_MAX in hadoop-env.sh</b><br>
+     <blockquote>New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we&apos;ve seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We&apos;ve observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
+<br>
+<br>Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we&apos;ve been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7146">HADOOP-7146</a>.
+     Major bug reported by tlipcon and fixed by tlipcon <br>
+     <b>RPC server leaks file descriptors</b><br>
+     <blockquote>Both the Listener and Responder thread call Selector.open but don&apos;t have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7145">HADOOP-7145</a>.
+     Major bug reported by tlipcon and fixed by tlipcon <br>
+     <b>Configuration.getLocalPath should trim whitespace from the provided directories</b><br>
+     <blockquote>MR and HDFS use the Configuration.getTrimmedStrings API for local directory lists, but in a few places also use Configuration.getLocalPath. The former API trims whitespace around each entry in the list, but the latter doesn&apos;t. This can cause some subtle problems - the latter API should be fixed to also trim the directory names.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7140">HADOOP-7140</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon <br>
+     <b>IPC Reader threads do not stop when server stops</b><br>
+     <blockquote>After HADOOP-6713, the new IPC &quot;Reader&quot; threads are not properly stopped when the server shuts down. One repercussion of this is that conditions that are supposed to shut down a daemon no longer work (eg the TT doesn&apos;t shut itself down if it detects an incompatible build version)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7137">HADOOP-7137</a>.
+     Major task reported by nidaley and fixed by nidaley <br>
+     <b>Remove hod contrib</b><br>
+     <blockquote>As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
+<br>svn remove common/trunk/src/contrib/hod
+<br>using this Jira.
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7134">HADOOP-7134</a>.
+     Major improvement reported by rvs and fixed by rvs (build)<br>
+     <b>configure files that are generated as part of the released tarball need to have executable bit set</b><br>
+     <blockquote>Currently the configure files that are packaged in a tarball are -rw-rw-r--</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7126">HADOOP-7126</a>.
+     Major bug reported by pocheung and fixed by pocheung <br>
+     <b>TestDFSShell fails in trunk</b><br>
+     <blockquote>TestDFSShell.testFilePermissions() fails on an assert in Windows.  This originated from HDFS-1084 but the fix is in Common.
+<br>
+<br>junit.framework.ComparisonFailure: null expected:&lt;rwxr[w----]&gt; but was:&lt;rwxr[-xr-x]&gt;
+<br>  at junit.framework.Assert.assertEquals(Assert.java:81)
+<br>  at junit.framework.Assert.assertEquals(Assert.java:87)
+<br>  at org.apache.hadoop.hdfs.TestDFSShell.confirmPermissionChange(TestDFSShell.java:836)
+<br>  at org.apache.hadoop.hdfs.TestDFSShell.testChmod(TestDFSShell.java:777)
+<br>  at org.apache.hadoop.hdfs.TestDFSShell.testFilePermissions(TestDFSShell.java:856)
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7122">HADOOP-7122</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon (util)<br>
+     <b>Timed out shell commands leak Timer threads</b><br>
+     <blockquote>When a shell command times out, the TimerThread used to cause the timeout is leaked.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7121">HADOOP-7121</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon (ipc)<br>
+     <b>Exceptions while serializing IPC call response are not handled well</b><br>
+     <blockquote>We had a situation where for some reason the serialization of an RPC call&apos;s response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7118">HADOOP-7118</a>.
+     Blocker bug reported by tlipcon and fixed by tlipcon <br>
+     <b>NPE in Configuration.writeXml</b><br>
+     <blockquote>In HADOOP-7082 I stupidly introduced a regression whereby Configuration.writeXml will throw an NPE if it is called before any .get() call is made, since the properties member is not initialized. This is causing a failure in TestCapacitySchedulerWithJobTracker on my box, but doesn&apos;t appear to trigger any failures in the non-contrib tests since .get() is usually called first.
+<br>
+<br>This JIRA is to fix the bug and add a unit test for writeXml in common (apparently it never had a unit test)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7110">HADOOP-7110</a>.
+     Major improvement reported by tlipcon and fixed by tlipcon (io, native)<br>
+     <b>Implement chmod with JNI</b><br>
+     <blockquote>MapReduce is currently using a race-prone workaround to approximate chmod() because forking chmod is too expensive. This race is causing build failures (and probably task failures too). We should implement chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7106">HADOOP-7106</a>.
+     Blocker improvement reported by nidaley and fixed by tlipcon (build)<br>
+     <b>Re-organize hadoop subversion layout</b><br>
+     <blockquote>As discussed on general@ at http://tinyurl.com/4q6lhxm</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7104">HADOOP-7104</a>.
+     Major bug reported by kzhang and fixed by kzhang (ipc, security)<br>
+     <b>Remove unnecessary DNS reverse lookups from RPC layer</b><br>
+     <blockquote>RPC connection authorization needs to verify client&apos;s Kerberos principal name matches what specified for the protocol. For service clients like DN&apos;s, their Kerberos principal names can be specified in the form of  &quot;datanode/_HOST@DOMAIN.COM&quot;. To get the expected
+<br>client principal name, the server needs to substitute &quot;_HOST&quot; with the client&apos;s fully qualified domain name, which requires a reverse DNS lookup from client IP address. However, for connections from clients whose principal name are either unspecified or specified not using the &quot;_HOST&quot; convention, the substitution is not required and the reverse DNS lookup should be avoided. Currently the reverse DNS lookup is done for all clients, which could slow services like NN down, when local named cache is not available.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7102">HADOOP-7102</a>.
+     Major bug reported by shv and fixed by shv (conf)<br>
+     <b>Remove &quot;fs.ramfs.impl&quot; field from core-deafult.xml</b><br>
+     <blockquote>&quot;fs.ramfs.impl&quot; used to be configuration parameter for InMemoryFileSystem, which was deprecated in 0.18 (HADOOP-3501) and removed in 0.21 (HADOOP-4648). Configuration should have been cleaned up at the time.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7101">HADOOP-7101</a>.
+     Blocker bug reported by tlipcon and fixed by tlipcon (security)<br>
+     <b>UserGroupInformation.getCurrentUser() fails when called from non-Hadoop JAAS context</b><br>
+     <blockquote>If a Hadoop client is run from inside a container like Tomcat, and the current AccessControlContext has a Subject associated with it that is not created by Hadoop, then UserGroupInformation.getCurrentUser() will throw NoSuchElementException, since it assumes that any Subject will have a hadoop User principal.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7100">HADOOP-7100</a>.
+     Blocker bug reported by tlipcon and fixed by tlipcon (build, contrib/cloud)<br>
+     <b>Build broken by HADOOP-6811</b><br>
+     <blockquote>The commit of HADOOP-6811 removed the ec2 contrib but didn&apos;t update build.xml, which references some of these files from the packaging targets. So, the hudson build is currently broken.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7097">HADOOP-7097</a>.
+     Blocker bug reported by nwatkins and fixed by nwatkins (build, native)<br>
+     <b>java.library.path missing basedir</b><br>
+     <blockquote>My Hadoop installation is  having trouble loading the native code library. It appears from the log below that java.library.path is missing the basedir in its path. The libraries are built, and present in the directory shown below (relative to hadoop-common directory). Instead of seeing:
+<br>
+<br> /build/native/Linux-amd64-64/lib
+<br>
+<br>I would expect to see:
+<br>
+<br> /path/to/hadoop-common/build/native/Linux-amd64-64/lib
+<br>
+<br>I&apos;m working in branch-0.22.
+<br>
+<br>2011-01-10 17:09:27,695 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
+<br>2011-01-10 17:09:27,695 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=/build/native/Linux-amd64-64/lib
+<br>2011-01-10 17:09:27,695 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7094">HADOOP-7094</a>.
+     Blocker bug reported by tlipcon and fixed by cos (build)<br>
+     <b>hadoop.css got lost during project split</b><br>
+     <blockquote>hadoop.css no longer exists in common or HDFS, so the web UIs look pretty ugly. The HTML still refers to this file, it&apos;s just gone.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7093">HADOOP-7093</a>.
+     Major bug reported by tlipcon and fixed by tlipcon (io)<br>
+     <b>Servlets should default to text/plain</b><br>
+     <blockquote>In trunk the servlets like /stacks and /metrics are returning text/html content-type instead of text/plain. Security wise it&apos;s much safer to default to text/plain and require servlets to explicitly set the content-type to text/html when required.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7091">HADOOP-7091</a>.
+     Major bug reported by kzhang and fixed by kzhang (security)<br>
+     <b>reloginFromKeytab() should happen even if TGT can&apos;t be found</b><br>
+     <blockquote>HADOOP-6965 introduced a getTGT() method and prevents reloginFromKeytab() from happening when TGT is not found. This results in the RPC layer not being able to refresh TGT after TGT expires. The reason is RPC layer only does relogin when the expired TGT is used and an exception is thrown. However, when that happens, the expired TGT will be removed from Subject. Therefore, getTGT() will return null and relogin will not be performed. We observed, for example, JT will not be able to re-connect to NN after TGT expires.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7089">HADOOP-7089</a>.
+     Minor bug reported by eli and fixed by eli (scripts)<br>
+     <b>Fix link resolution logic in hadoop-config.sh</b><br>
+     <blockquote>The link resolution logic in bin/hadoop-config.sh fails when when executed via a symlink, from the root directory.  We can replace this logic with cd -P and pwd -P, which should be portable across Linux, Solaris, BSD, and OSX. </blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7087">HADOOP-7087</a>.
+     Blocker bug reported by tlipcon and fixed by tlipcon (io)<br>
+     <b>SequenceFile.createWriter ignores FileSystem parameter</b><br>
+     <blockquote>The SequenceFile.createWriter methods that take a FileSystem ignore this parameter after HADOOP-6856. This is causing some MR tests to fail and is a breaking change when users pass unqualified paths to these calls.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7082">HADOOP-7082</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon (conf)<br>
+     <b>Configuration.writeXML should not hold lock while outputting</b><br>
+     <blockquote>Common side of HDFS-1542</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7070">HADOOP-7070</a>.
+     Critical bug reported by tlipcon and fixed by tlipcon (security)<br>
+     <b>JAAS configuration should delegate unknown application names to pre-existing configuration</b><br>
+     <blockquote>As reported here: https://issues.cloudera.org/browse/DISTRO-66 it is impossible to use secured Hadoop inside an application that relies on other JAAS configurations. This is because the static initializer of UserGroupInformation replaces the JAAS configuration, but we don&apos;t delegate unknown applications up to whatever Configuration was installed previously. The delegation technique seems to be used by JBoss&apos;s XMLLoginConfigImpl for example.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7068">HADOOP-7068</a>.
+     Major bug reported by vicaya and fixed by vicaya <br>
+     <b>Ivy resolve force mode should be turned off by default</b><br>
+     <blockquote>The problem is introduced by  HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people&apos;s) time to track down the issue.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7057">HADOOP-7057</a>.
+     Minor bug reported by cos and fixed by cos (util)<br>
+     <b>IOUtils.readFully and IOUtils.skipFully have typo in exception creation&apos;s message</b><br>
+     <blockquote>throw new IOException( &quot;Premeture EOF from inputStream&quot;);
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7054">HADOOP-7054</a>.
+     Major improvement reported by sanjay.radia and fixed by sanjay.radia <br>
+     <b>Change NN LoadGenerator to use the new FileContext api</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7046">HADOOP-7046</a>.
+     Blocker bug reported by nidaley and fixed by pocheung (security)<br>
+     <b>1 Findbugs warning on trunk and branch-0.22</b><br>
+     <blockquote>There is 1 findbugs warnings on trunk. See attached html file. This must be fixed or filtered out to get back to 0 warnings. The OK_FINDBUGS_WARNINGS property in src/test/test-patch.properties should also be set to 0 in the patch that fixes this issue.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7038">HADOOP-7038</a>.
+     Minor bug reported by gnawux and fixed by gnawux (build)<br>
+     <b>saveVersion script includes an additional \r while running whoami under windows</b><br>
+     <blockquote>I built common under windows occasionally, and found it failed because the &apos;user&apos; in build/src/o/a/h/package-info.java is &quot;myhostmyname^M&quot;.
+<br>It seems because the whoami of windows give a string with &apos;\n\r&apos; rather than &apos;\n&apos; only. thus I add an additional tr for it to eliminate the problem.
+<br>Since only windows would generate &apos;\n\r&apos; output, I think it won&apos;t harm to any other platform.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7034">HADOOP-7034</a>.
+     Major test reported by eli and fixed by eli (fs)<br>
+     <b>Add TestPath tests to cover dot, dot dot, and slash normalization</b><br>
+     <blockquote>Add tests for the current path normalization for dot, dot dot, and slash in TestPath (from HDFS-836).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7032">HADOOP-7032</a>.
+     Major improvement reported by eli and fixed by eli (fs)<br>
+     <b>Assert type constraints in the FileStatus constructor</b><br>
+     <blockquote>A FileStatus may represent a file, directory or symlink.  This is indicated using the isdir and symlink members, let&apos;s add an assert that validates the contstraints on these members (eg a directory may not have the symlink member set).  We could also verify this by having more than one constructor but we don&apos;t statically know the type of the file status when we create it.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7028">HADOOP-7028</a>.
+     Minor bug reported by patrickangeles and fixed by patrickangeles (build)<br>
+     <b>ant eclipse does not include requisite ant.jar in the classpath</b><br>
+     <blockquote>RccTask imports classes from ant.jar
+<br>Importing the project into Eclipse results in these classes not being found, because ant.jar is not in the classpath.
+<br>Note that this patch requires that ANT_HOME be set, but this is consistent with the documentation as per
+<br>http://wiki.apache.org/hadoop/EclipseEnvironment
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7024">HADOOP-7024</a>.
+     Major test reported by kzhang and fixed by kzhang (test)<br>
+     <b>Create a test method for adding file systems during tests.</b><br>
+     <blockquote>It allows a (mocked) filesystem object to be added to cache for testing purposes. This is used by HDFS-1187.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7013">HADOOP-7013</a>.
+     Major improvement reported by pkling and fixed by pkling <br>
+     <b>Add boolean field isCorrupt to BlockLocation</b><br>
+     <blockquote>This is needed to allow DFSClient.getBlockLocations to notify the calling application when returning a BlockLocation that corresponds to a corrupt block. Currently, this happens when there are no uncorrupted replicas of a requested block.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7011">HADOOP-7011</a>.
+     Major bug reported by atm and fixed by atm <br>
+     <b>KerberosName.main(...) throws NPE</b><br>
+     <blockquote>The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7010">HADOOP-7010</a>.
+     Minor improvement reported by yaojingguo and fixed by yaojingguo (fs)<br>
+     <b>Typo in FileSystem.java</b><br>
+     <blockquote>For the Javadoc for getLocal method, &quot;Get the local file syste&quot; should be &quot;Get the local file system.&quot;.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7009">HADOOP-7009</a>.
+     Major improvement reported by hairong and fixed by hairong (io)<br>
+     <b>MD5Hash provides a public factory method that creates an instance of MessageDigest</b><br>
+     <blockquote>MD5Hash has a private way of creating a MessageDigest object that&apos;s thread local. I&apos;d like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7008">HADOOP-7008</a>.
+     Major improvement reported by nidaley and fixed by gkesavan (test)<br>
+     <b>Enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings</b><br>
+     <blockquote>test-patch.sh should be able to accept a properties file containing an acceptable number of findbugs and javadoc warnings.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7007">HADOOP-7007</a>.
+     Major improvement reported by gkesavan and fixed by gkesavan (build)<br>
+     <b>update the hudson-test-patch target to work with the latest test-patch script.</b><br>
+     <blockquote>The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7006">HADOOP-7006</a>.
+     Major bug reported by cnauroth and fixed by cnauroth (fs)<br>
+     <b>hadoop fs -getmerge does not work using codebase from trunk.</b><br>
+     <blockquote>Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7005">HADOOP-7005</a>.
+     Major improvement reported by nidaley and fixed by nidaley (test)<br>
+     <b>Update test-patch.sh to remove callback to Hudson master</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6996">HADOOP-6996</a>.
+     Major new feature reported by hairong and fixed by hairong (io)<br>
+     <b>Allow CodecFactory to return a codec object given a codec&apos; class name</b><br>
+     <blockquote>CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file&apos;s name. I would like to make getCodec method to alternatively take a codec&apos;s class name.
+<br>
+<br>This is required by  HDFS-1435, where
+<br>1. it allows an HDFS admin to configure which codec to use to save an image. 
+<br>2. It stores the codec class name in its on-disk image instead of a file&apos;s suffix.
+<br>
+<br>When saving and reading an image, I&apos;d like to get an codec from CodecFactory by its class name. </blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6991">HADOOP-6991</a>.
+     Minor bug reported by chris.douglas and fixed by chris.douglas <br>
+     <b>SequenceFile::Reader discards length for files, does not call openFile</b><br>
+     <blockquote>While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6989">HADOOP-6989</a>.
+     Major bug reported by jghoman and fixed by chris.douglas <br>
+     <b>TestSetFile is failing on trunk</b><br>
+     <blockquote>Testsuite: org.apache.hadoop.io.TestSetFile
+<br>Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
+<br>------------- Standard Output ---------------
+<br>2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
+<br>2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
+<br>2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
+<br>------------- ---------------- ---------------
+<br>
+<br>Testcase: testSetFile took 0.964 sec
+<br>  Caused an ERROR
+<br>key class or comparator option must be set
+<br>java.lang.IllegalArgumentException: key class or comparator option must be set
+<br>  at org.apache.hadoop.io.MapFile$Writer.&lt;init&gt;(MapFile.java:247)
+<br>  at org.apache.hadoop.io.SetFile$Writer.&lt;init&gt;(SetFile.java:60)
+<br>  at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
+<br>  at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6987">HADOOP-6987</a>.
+     Major improvement reported by jghoman and fixed by jghoman (test)<br>
+     <b>Use JUnit Rule to optionally fail test cases that run more than 10 seconds</b><br>
+     <blockquote>Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6985">HADOOP-6985</a>.
+     Minor improvement reported by rvadali and fixed by rvadali <br>
+     <b>Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template</b><br>
+     <blockquote>For an administrator who wants to customize HADOOP_OPTS, it would be better to have
+<br>
+<br># if [ &quot;$HADOOP_OPTS&quot; == &quot;&quot; ]; then export HADOOP_OPTS=-server; else FOO+=&quot; -server&quot;; fi
+<br>
+<br>instead of
+<br>
+<br># Extra Java runtime options.  Empty by default.
+<br># export HADOOP_OPTS=-server
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6984">HADOOP-6984</a>.
+     Minor bug reported by chris.douglas and fixed by chris.douglas (io)<br>
+     <b>NPE from SequenceFile::Writer.CompressionCodecOption</b><br>
+     <blockquote>The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6978">HADOOP-6978</a>.
+     Blocker new feature reported by tlipcon and fixed by tlipcon (io, native, security)<br>
+     <b>Add JNI support for secure IO operations</b><br>
+     <blockquote>In support of MAPREDUCE-2096, we need to add some JNI functionality. In particular, we need the ability to use fstat() on an open file stream, and to use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6977">HADOOP-6977</a>.
+     Major improvement reported by cos and fixed by cos (test)<br>
+     <b>Herriot daemon clients should vend statistics</b><br>
+     <blockquote>The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
+<br>
+<br>The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6975">HADOOP-6975</a>.
+     Major bug reported by pkling and fixed by pkling <br>
+     <b>integer overflow in S3InputStream for blocks &gt; 2GB</b><br>
+     <blockquote>S3InputStream has the same integer overflow issue as DFSInputStream (fixed in HDFS-96).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6970">HADOOP-6970</a>.
+     Major bug reported by shv and fixed by boryas (build)<br>
+     <b>SecurityAuth.audit should be generated under /build</b><br>
+     <blockquote>SecurityAuth.audit is generated under currently root project directory whenever I run anything, and is not being cleaned up by the clean target. It should be created under build directory instead.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6965">HADOOP-6965</a>.
+     Major bug reported by jnp and fixed by jnp <br>
+     <b>Method in UGI to get Kerberos ticket. </b><br>
+     <blockquote>The getTGT method in AutoRenewal thread is moved to the outer UGI class. It is still a private method but can be used by reloginFromKeyTab to check for TGT expiry. This jira covers common changes for HDFS-1364</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6951">HADOOP-6951</a>.
+     Major bug reported by atm and fixed by atm (security)<br>
+     <b>Distinct minicluster services (e.g. NN and JT) overwrite each other&apos;s service policies</b><br>
+     <blockquote>Because the protocol -&gt; ACL mapping in ServiceAuthorizationManager is static, services which are run in the same JVM have the potential to clobber the other&apos;s service authorization ACLs whenever ServiceAuthorizationManager.refresh() is called. This causes authorization failures if one tries to launch a 2NN connected to a minicluster with hadoop.security.authorization enabled. Seems like each service should have its own instance of a ServiceAuthorizationManager, instead of using static methods.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6950">HADOOP-6950</a>.
+     Trivial improvement reported by philip and fixed by philip (scripts)<br>
+     <b>Suggest that HADOOP_CLASSPATH should be preserved in hadoop-env.sh.template</b><br>
+     <blockquote>HADOOP_CLASSPATH tends to be used to add to bin/hadoop&apos;s classpath.  Because of the way the comment is written, administrator&apos;s who customize hadoop-env.sh often inadvertently disable user&apos;s abilities to use it, by not including the present value of the variable.
+<br>
+<br>I propose we change the commented out suggestion code to include the present value.
+<br>
+<br> # Extra Java CLASSPATH elements.  Optional.
+<br>-# export HADOOP_CLASSPATH=
+<br>+# export HADOOP_CLASSPATH=&quot;&lt;extra_entries&gt;:$HADOOP_CLASSPATH&quot;
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6949">HADOOP-6949</a>.
+     Major improvement reported by navis and fixed by mattf (io)<br>
+     <b>Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting</b><br>
+     <blockquote>Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
+<br>
+<br>It would not be needed to specify each element types for primitive arrays.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6947">HADOOP-6947</a>.
+     Major bug reported by tlipcon and fixed by tlipcon (security)<br>
+     <b>Kerberos relogin should set refreshKrb5Config to true</b><br>
+     <blockquote>In working on securing a daemon that uses two different principals from different threads, I found that I wasn&apos;t able to login from a second keytab after I&apos;d logged in from the first. This is because we don&apos;t set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won&apos;t switch over to the correct keytab file if it&apos;s different than the first.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6943">HADOOP-6943</a>.
+     Major improvement reported by atm and fixed by atm (security)<br>
+     <b>The GroupMappingServiceProvider interface should be public</b><br>
+     <blockquote>The GroupMappingServiceProvider interface is presently package-protected. It seems likely that many organizations will be implementing their own versions of this to suit their particular setup. It would be nice if this interface were made public, and annotated with &quot;@InterfaceAudience.Private&quot; and &quot;@InterfaceStability.Evolving&quot;.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6940">HADOOP-6940</a>.
+     Minor bug reported by tomwhite and fixed by tomwhite (fs)<br>
+     <b>RawLocalFileSystem&apos;s markSupported method misnamed markSupport</b><br>
+     <blockquote>It should be named markSupported to override the method defined in InputStream. Since it doesn&apos;t change the default no harm is done.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6938">HADOOP-6938</a>.
+     Major bug reported by kzhang and fixed by kzhang (ipc, security)<br>
+     <b>ConnectionId.getRemotePrincipal() should check if security is enabled</b><br>
+     <blockquote>When security is not enabled, getRemotePrincipal() should return null, which means the Kerberos principal of the remote server is ignored. This bug was caught by TestCLI on Yahoo 20S branch.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6933">HADOOP-6933</a>.
+     Minor bug reported by tlipcon and fixed by tlipcon (test)<br>
+     <b>TestListFiles is flaky</b><br>
+     <blockquote>TestListFiles assumes a particular order of the files returned by the directory iterator. There&apos;s no such guarantee made by the underlying API, so the test fails on some hosts.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6932">HADOOP-6932</a>.
+     Major bug reported by boryas and fixed by boryas <br>
+     <b>Namenode start (init) fails because of invalid kerberos key, even when security set to &quot;simple&quot;</b><br>
+     <blockquote>NameNode.initialize() calls login() method even when security set to simple</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6930">HADOOP-6930</a>.
+     Major bug reported by sharadag and fixed by sharadag (ipc)<br>
+     <b>AvroRpcEngine doesn&apos;t work with generated Avro code</b><br>
+     <blockquote>AvroRpcEngine uses &apos;reflect&apos; based java implementation. There should be a way to have it work with &apos;specific&apos; (generated code from avro idl).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6926">HADOOP-6926</a>.
+     Minor bug reported by tlipcon and fixed by tlipcon (io)<br>
+     <b>SocketInputStream incorrectly implements read()</b><br>
+     <blockquote>SocketInputStream&apos;s read() implementation doesn&apos;t upcast to int correctly, so it can&apos;t read bytes &gt; 0x80. This is the same bug as HADOOP-6925, but in a different spot.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6922">HADOOP-6922</a>.
+     Major bug reported by ravidotg and fixed by ravidotg (documentation, security)<br>
+     <b>COMMON part of MAPREDUCE-1664</b><br>
+     <blockquote>MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6913">HADOOP-6913</a>.
+     Major bug reported by kzhang and fixed by kzhang (security)<br>
+     <b>Circular initialization between UserGroupInformation and KerberosName</b><br>
+     <blockquote>If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName&apos;s static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn&apos;t been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6911">HADOOP-6911</a>.
+     Minor improvement reported by boryas and fixed by  <br>
+     <b>doc update for DelegationTokenFetcher (part of HDFS-1036)</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6907">HADOOP-6907</a>.
+     Major bug reported by kzhang and fixed by kzhang (ipc, security)<br>
+     <b>Rpc client doesn&apos;t use the per-connection conf to figure out server&apos;s Kerberos principal</b><br>
+     <blockquote>Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server&apos;s Kerberos principal on a per-connection basis. It&apos;s not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6906">HADOOP-6906</a>.
+     Major bug reported by vinodkv and fixed by vinodkv (fs)<br>
+     <b>FileContext copy() utility doesn&apos;t work with recursive copying of directories.</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6905">HADOOP-6905</a>.
+     Major improvement reported by kzhang and fixed by kzhang (security)<br>
+     <b>Better logging messages when a delegation token is invalid</b><br>
+     <blockquote>From our production logs, we see some logging messages of &quot;token is expired or doesn&apos;t exist&quot;. It would be helpful to know whose token it was.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6903">HADOOP-6903</a>.
+     Major improvement reported by sanjay.radia and fixed by sanjay.radia <br>
+     <b>Make AbstractFileSystem&apos;s methods public to allow filter-Fs like implementions in a differnt package than fs</b><br>
+     <blockquote>Make AbstractFileSystem&apos;s methods public to allow filter-Fs like implementions in a differnt package than fs</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6900">HADOOP-6900</a>.
+     Major bug reported by sureshms and fixed by hairong <br>
+     <b>FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions</b><br>
+     <blockquote>HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6899">HADOOP-6899</a>.
+     Major bug reported by sanjay.radia and fixed by sanjay.radia (fs)<br>
+     <b>RawLocalFileSystem#setWorkingDir() does not work for relative names</b><br>
+     <blockquote>RawLocalFileSystem#setWorkingDir() does not work for relative names</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6898">HADOOP-6898</a>.
+     Blocker bug reported by tlipcon and fixed by atm (fs, security)<br>
+     <b>FileSystem.copyToLocal creates files with 777 permissions</b><br>
+     <blockquote>FileSystem.copyToLocal ends up calling through to FileUtil.copy, which calls create() on the target file system without passing any permission object. Therefore, the file ends up getting created locally with 777 permissions, which is dangerous -- even if the caller then fixes up permissions afterwards, it exposes a window in which an attacker can open the file.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6892">HADOOP-6892</a>.
+     Major new feature reported by jghoman and fixed by jghoman (security)<br>
+     <b>Common component of HDFS-1150 (Verify datanodes&apos; identities to clients in secure clusters)</b><br>
+     <blockquote>HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6890">HADOOP-6890</a>.
+     Major improvement reported by hairong and fixed by hairong (fs)<br>
+     <b>Improve listFiles API introduced by HADOOP-6870</b><br>
+     <blockquote>This jira is mainly for addressing Suresh&apos;s review comments for HADOOP-6870:
+<br>
+<br>   1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
+<br>   2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
+<br>   3. FileSystem.java
+<br>          * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
+<br>          * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
+<br>   4. TestListFiles.java
+<br>          * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6889">HADOOP-6889</a>.
+     Major new feature reported by hairong and fixed by johnvijoe (ipc)<br>
+     <b>Make RPC to have an option to timeout</b><br>
+     <blockquote>Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
+<br>
+<br>But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica&apos;s length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
+<br>
+<br>I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6888">HADOOP-6888</a>.
+     Major bug reported by kzhang and fixed by kzhang (fs, security)<br>
+     <b>Being able to close all cached FileSystem objects for a given UGI</b><br>
+     <blockquote>This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6885">HADOOP-6885</a>.
+     Major bug reported by eli and fixed by eli (security)<br>
+     <b>Fix java doc warnings in Groups and RefreshUserMappingsProtocol</b><br>
+     <blockquote>There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6884">HADOOP-6884</a>.
+     Major improvement reported by zasran and fixed by zasran <br>
+     <b>Add LOG.isDebugEnabled() guard for each LOG.debug(&quot;...&quot;)</b><br>
+     <blockquote>Each LOG.debug(&quot;...&quot;) should be executed only if LOG.isDebugEnabled() is true, in some cases it&apos;s expensive to construct the string that is being printed to log. It&apos;s much easier to always use LOG.isDebugEnabled() because it&apos;s easier to check (rather than in each case reason whether it&apos;s necessary or not).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6879">HADOOP-6879</a>.
+     Major improvement reported by iyappans and fixed by cos (build, test)<br>
+     <b>Provide SSH based (Jsch) remote execution API for system tests</b><br>
+     <blockquote>http://mvnrepository.com/
+<br>com.jcraft ยป jsch 
+<br>0.1.42 version needs to be included in the build. This is  needed to facilitate implementation of some system (Herriot) testcases .
+<br>
+<br>Please include this in ivy.
+<br>
+<br>jsch is originally located in http://www.jcraft.com/jsch/</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6877">HADOOP-6877</a>.
+     Major improvement reported by kzhang and fixed by kzhang (ipc)<br>
+     <b>Common part of HDFS-1178</b><br>
+     <blockquote>This is the Common part of HDFS-1178.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6873">HADOOP-6873</a>.
+     Major bug reported by boryas and fixed by boryas <br>
+     <b>using delegation token over hftp for long running clients (part of hdfs 1296)</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6870">HADOOP-6870</a>.
+     Major new feature reported by hairong and fixed by hairong (fs)<br>
+     <b>Add FileSystem#listLocatedStatus to list a directory&apos;s content together with each file&apos;s block locations</b><br>
+     <blockquote>This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual &quot;getFileBlockLocations&quot; calls to NN for each file in the input directory of a job. Instead, a file&apos;s block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6862">HADOOP-6862</a>.
+     Major improvement reported by amareshwari and fixed by amareshwari (security)<br>
+     <b>Add api to add user/group to AccessControlList</b><br>
+     <blockquote>Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6861">HADOOP-6861</a>.
+     Major improvement reported by jnp and fixed by jnp <br>
+     <b>Method in Credentials to read and write a token storage file.</b><br>
+     <blockquote>The jira covers the changes in common corresponding to MAPREDUCE-1566. 
+<br>
+<br>This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6859">HADOOP-6859</a>.
+     Major improvement reported by sureshms and fixed by sureshms <br>
+     <b>Introduce additional statistics to FileSystem</b><br>
+     <blockquote>Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6856">HADOOP-6856</a>.
+     Major improvement reported by owen.omalley and fixed by owen.omalley (io)<br>
+     <b>SequenceFile and MapFile need cleanup to remove redundant constructors</b><br>
+     <blockquote>Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don&apos;t cover the entire space.
+<br>
+<br>All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
+<br>
+<br>I&apos;d like change to the style of FileContext.create with option parameters. I&apos;ll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I&apos;ll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6853">HADOOP-6853</a>.
+     Major bug reported by jghoman and fixed by jghoman <br>
+     <b>Common component of HDFS-1045</b><br>
+     <blockquote>HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6846">HADOOP-6846</a>.
+     Major task reported by tomwhite and fixed by tomwhite (build)<br>
+     <b>Scripts for building Hadoop 0.22.0 release</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6845">HADOOP-6845</a>.
+     Major improvement reported by jnp and fixed by jnp <br>
+     <b>TokenStorage renamed to Credentials.</b><br>
+     <blockquote>This jira tracks common changes for MAPREDUCE-1528.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6835">HADOOP-6835</a>.
+     Major improvement reported by tomwhite and fixed by roelofs (io)<br>
+     <b>Support concatenated gzip files</b><br>
+     <blockquote>When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6834">HADOOP-6834</a>.
+     Major bug reported by ahadr and fixed by hong.tang (io)<br>
+     <b>TFile.append compares initial key against null lastKey  </b><br>
+     <blockquote>The following code in TFile.KeyReigster.close: 
+<br>
+<br>            byte[] lastKey = lastKeyBufferOS.getBuffer();
+<br>            int lastLen = lastKeyBufferOS.size();
+<br>            if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
+<br>                lastLen) &lt; 0) {
+<br>              throw new IOException(&quot;Keys are not added in sorted order&quot;);
+<br>            }
+<br>
+<br>compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable&apos;s comparator, which ignores length and thus produces incorrect results. 
+<br></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6832">HADOOP-6832</a>.
+     Major new feature reported by owen.omalley and fixed by owen.omalley (security)<br>
+     <b>Provide a web server plugin that uses a static user for the web UI</b><br>
+     <blockquote>We need a simple plugin that uses a static user for clusters with security that don&apos;t want to authenticate users on the web UI.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6825">HADOOP-6825</a>.
+     Major improvement reported by rschmidt and fixed by rschmidt <br>
+     <b>FileStatus needs unit tests</b><br>
+     <blockquote>We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6818">HADOOP-6818</a>.
+     Major improvement reported by devaraj and fixed by devaraj (security)<br>
+     <b>Provide a JNI-based implementation of GroupMappingServiceProvider</b><br>
+     <blockquote>The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6815">HADOOP-6815</a>.
+     Major bug reported by boryas and fixed by boryas <br>
+     <b>refreshSuperUserGroupsConfiguration should use server side configuration for the refresh</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6814">HADOOP-6814</a>.
+     Major improvement reported by jnp and fixed by jnp <br>
+     <b>Method in UGI to get the authentication method of the real user.</b><br>
+     <blockquote>UGI should have a method to return the authentication method of the real user for a proxy-user scenario.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6812">HADOOP-6812</a>.
+     Major bug reported by appodictic and fixed by chris.douglas (documentation)<br>
+     <b>fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.</b><br>
+     <blockquote>http://hadoop.apache.org/common/docs/current/cluster_setup.html
+<br>fs.inmemory.size.mb does not appear in any xml file
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6811">HADOOP-6811</a>.
+     Blocker improvement reported by tomwhite and fixed by tomwhite <br>
+     <b>Remove EC2 bash scripts</b><br>
+     <blockquote>The bash scripts are deprecated in 0.21 (HADOOP-6403) in favour of scripts in Whirr (http://incubator.apache.org/projects/whirr.html). They should be removed in 0.22. </blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6805">HADOOP-6805</a>.
+     Major improvement reported by boryas and fixed by boryas <br>
+     <b>add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6803">HADOOP-6803</a>.
+     Major test reported by eli and fixed by eli (io)<br>
+     <b>Add native gzip read/write coverage to TestCodec </b><br>
+     <blockquote>Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
+<br>
+<br>      while (!compressor.finished()) {
+<br>        compress();
+<br>      }
+<br>
+<br>I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it&apos;s been finished.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6791">HADOOP-6791</a>.
+     Major improvement reported by boryas and fixed by boryas <br>
+     <b>Refresh for proxy superuser config  (common part for HDFS-1096)</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6787">HADOOP-6787</a>.
+     Major bug reported by vicaya and fixed by vicaya (fs)<br>
+     <b>Factor out glob pattern code from FileContext and Filesystem</b><br>
+     <blockquote>Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
+<br>
+<br>Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6781">HADOOP-6781</a>.
+     Major bug reported by boryas and fixed by boryas <br>
+     <b>security audit log shouldn&apos;t have exception in it.</b><br>
+     <blockquote>security audit log in Server.java also prints the exception information. It shouldn&apos;t be there.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6778">HADOOP-6778</a>.
+     Major bug reported by boryas and fixed by boryas <br>
+     <b>add isRunning() method to AbstractDelegationTokenSecretManager (for  HDFS-1044)</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6763">HADOOP-6763</a>.
+     Major bug reported by owen.omalley and fixed by boryas <br>
+     <b>Remove verbose logging from the Groups class</b><br>
+     <blockquote>
+2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:&lt;init&gt;(60)) - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
+<br>...
+<br>2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Returning cached groups for &apos;oom&apos;
+<br>
+<br>should both be demoted to debug level.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6761">HADOOP-6761</a>.
+     Major improvement reported by dms and fixed by dms <br>
+     <b>Improve Trash Emptier</b><br>
+     <blockquote>There are two inefficiencies in the Trash functionality right now that have caused some problems for us.
+<br>First if you configured your trash interval to be one day (24 hours) that means that you store 2 days worth of data eventually. The Current and the previous timestamp that will not be deleted until the end of the interval.
+<br>And another problem is accumulating a lot of data in Trash before the Emptier wakes up. If there are a couple of million files trashed and the Emptier does deletion on HDFS the NameNode will freeze until everything is removed. (this particular problem hopefully will be addressed with HDFS-1143).
+<br>
+<br>My proposal is to have two configuration intervals. One for deleting the trashed data and another for checkpointing. This way for example for intervals of one day and one hour we will only store 25 hours of data instead of 48 right now and the deletions will be happening in smaller chunks every hour of the day instead of a huge deletion at the end of the day now.
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6758">HADOOP-6758</a>.
+     Major bug reported by azaroth and fixed by azaroth <br>
+     <b>MapFile.fix does not allow index interval definition</b><br>
+     <blockquote>When using the static methond MapFile.fix() there is no way to override the default IndexInterval that is 128.
+<br>
+<br>The IndexInterval should be taken from the configuration that is passed to the method.
+<br>int indexInterval = 128; 
+<br>indexInterval = conf.getInt(INDEX_INTERVAL, indexInterval); 
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6756">HADOOP-6756</a>.
+     Major bug reported by zasran and fixed by zasran (fs)<br>
+     <b>Clean up and add documentation for configuration keys in CommonConfigurationKeys.java</b><br>
+     <blockquote>Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6747">HADOOP-6747</a>.
+     Major bug reported by vicaya and fixed by tlipcon <br>
+     <b>TestNetUtils fails on Mac OS X</b><br>
+     <blockquote>TestNetUtils fails consistently after HADOOP-6722 on Mac OS X Leopard 10.5.8:
+<br>
+<br>------------- Standard Error -----------------
+<br>local address: /127.0.0.1
+<br>local port: 64991
+<br>------------- ---------------- ---------------
+<br>
+<br>Testcase: testAvoidLoopbackTcpSockets took 0.421 sec
+<br>        Caused an ERROR
+<br>Invalid argument
+<br>java.net.SocketException: Invalid argument
+<br>        at sun.nio.ch.Net.connect(Native Method)
+<br>        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
+<br>        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
+<br>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:369)
+<br>        at org.apache.hadoop.net.TestNetUtils.testAvoidLoopbackTcpSockets(TestNetUtils.java:46)
+<br>
+<br>Although TCP spec seems to allow it, at least one implementation disallows this corner case. 
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6745">HADOOP-6745</a>.
+     Minor improvement reported by boryas and fixed by boryas (ipc)<br>
+     <b>adding some java doc to Server.RpcMetrics, UGI</b><br>
+     <blockquote></blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6730">HADOOP-6730</a>.
+     Major bug reported by eli and fixed by raviphulari (fs, test)<br>
+     <b>Bug in FileContext#copy and provide base class for FileContext tests</b><br>
+     <blockquote>Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
+<br>On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
+<br> *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
+</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6715">HADOOP-6715</a>.
+     Major bug reported by ravidotg and fixed by ravidotg (security, util)<br>
+     <b>AccessControlList.toString() returns empty string when we set acl to &quot;*&quot;</b><br>
+     <blockquote>AccessControlList.toString() returns empty string when we set the acl to &quot;\*&quot; and also when we set it to empty(i.e. &quot; &quot;). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to &quot;\*&quot;.
+<br>
+<br>I think AccessControlList.toString() needs to be changed to return &quot;\*&quot; when we set the acl to &quot;\*&quot;.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6714">HADOOP-6714</a>.
+     Major improvement reported by patrickangeles and fixed by patrickangeles <br>
+     <b>FsShell &apos;hadoop fs -text&apos; does not support compression codecs</b><br>
+     <blockquote>Currently, &apos;hadoop fs -text myfile&apos; looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means &apos;fs -text&apos; cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).
+<br>
+<br>It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6706">HADOOP-6706</a>.
+     Major bug reported by devaraj and fixed by devaraj (security)<br>
+     <b>Relogin behavior for RPC clients could be improved</b><br>
+     <blockquote>Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client&apos;s ticket is invalid). This jira is to fix that behavior.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6693">HADOOP-6693</a>.
+     Major improvement reported by sureshms and fixed by sureshms <br>
+     <b>Add metrics to track kerberos login activity</b><br>
+     <blockquote>Need metrics to track kerberos login activity such as login rate and the time taken for login.</blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6683">HADOOP-6683</a>.
+     Minor sub-task reported by xiaokang and fixed by xiaokang (io)<br>
+     <b>the first optimization: ZlibCompressor does not fully utilize the buffer</b><br>
+     <blockquote>Thanks for Hong Tang&apos;s advice.
+<br>
+<br>Sub task created for the first optimization. HADOOP-6662 closed. </blockquote></li>
+
+<li> <a href="https://issues.apache.org/jira/browse/HADOOP-6682">HADOOP-6682</a>.
+     Major bug reported by jghoman and fixed by jghoman (io)<br>
+     <b>NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly</b><br>
+     <blockquote>  public static String normalizeHostName(String name) {
+<br>    if (Character.digit(name.charAt(0), 16) != -1) {
+<br>      return name;
+<br>

[... 3727 lines stripped ...]


Mime
View raw message