Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EA7A77841 for ; Thu, 29 Sep 2011 00:19:20 +0000 (UTC) Received: (qmail 7396 invoked by uid 500); 29 Sep 2011 00:19:20 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 7364 invoked by uid 500); 29 Sep 2011 00:19:20 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 7356 invoked by uid 99); 29 Sep 2011 00:19:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Sep 2011 00:19:20 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Sep 2011 00:19:16 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 4AC8A23888CC for ; Thu, 29 Sep 2011 00:18:55 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1177119 - in /hadoop/common/branches/branch-0.20-security-205: CHANGES.txt src/docs/releasenotes.html Date: Thu, 29 Sep 2011 00:18:55 -0000 To: common-commits@hadoop.apache.org From: mattf@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20110929001855.4AC8A23888CC@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: mattf Date: Thu Sep 29 00:18:54 2011 New Revision: 1177119 URL: http://svn.apache.org/viewvc?rev=1177119&view=rev Log: Preparing for release 0.20.205.0 (rc1) Modified: hadoop/common/branches/branch-0.20-security-205/CHANGES.txt hadoop/common/branches/branch-0.20-security-205/src/docs/releasenotes.html Modified: hadoop/common/branches/branch-0.20-security-205/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-205/CHANGES.txt?rev=1177119&r1=1177118&r2=1177119&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20-security-205/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.20-security-205/CHANGES.txt Thu Sep 29 00:18:54 2011 @@ -1,6 +1,6 @@ Hadoop Change Log -Release 0.20.205.0 - 2011.09.27 +Release 0.20.205.0 - 2011.09.28 NEW FEATURES Modified: hadoop/common/branches/branch-0.20-security-205/src/docs/releasenotes.html URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-205/src/docs/releasenotes.html?rev=1177119&r1=1177118&r2=1177119&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20-security-205/src/docs/releasenotes.html (original) +++ hadoop/common/branches/branch-0.20-security-205/src/docs/releasenotes.html Thu Sep 29 00:18:54 2011 @@ -17,6 +17,7 @@

Changes since Hadoop 0.20.204.0

    +
  • HADOOP-6722. Major bug reported by tlipcon and fixed by tlipcon (util)
    NetUtils.connect should check that it hasn't connected a socket to itself
    @@ -67,6 +68,11 @@ RPC client should deal with the IP address changes
    The current RPC client implementation and the client-side callers assume that the hostname-address mappings of servers never change. The resolved address is stored in an immutable InetSocketAddress object above/outside RPC, and the reconnect logic in the RPC Connection implementation also trusts the resolved address that was passed down.

    If the NN suffers a failure that requires migration, it may be started on a different node with a different IP address. In this case, even if the name-addre...
    • +
  • HADOOP-7510. + Major improvement reported by daryn and fixed by daryn (security)
    + Tokens should use original hostname provided instead of ip
    +
    Tokens currently store the ip:port of the remote server. This precludes tokens from being used after a host's ip is changed. Tokens should store the hostname used to make the RPC connection. This will enable new processes to use their existing tokens.
    • +
  • HADOOP-7539. Major bug reported by johnvijoe and fixed by johnvijoe
    merge hadoop archive goodness from trunk to .20
    @@ -187,6 +193,26 @@ log4j.properties is missing properties for security audit and hdfs audit should be changed to info
    log4j.properties defines the security audit and hdfs audit files but is missing properties for security audit which causes security audit logs to not be present and also updates the hdfs audit to log at a WARN level. hdfs-audit logs should be at the INFO level so admin's/users can track when the namespace got the appropriate change.
    • +
  • HADOOP-7683. + Minor bug reported by arpitgupta and fixed by arpitgupta
    + hdfs-site.xml template has properties that are not used in 20
    +
    properties dfs.namenode.http-address and dfs.namenode.https-address should be removed
    • + +
  • HADOOP-7684. + Major bug reported by eyang and fixed by eyang (scripts)
    + jobhistory server and secondarynamenode should have init.d script
    +
    The current set of init.d scripts can start/stop process for:

    namenode
    datanode
    jobtracker
    tasktracker

    It is missing init.d scripts for:

    secondarynamenode
    jobhistory
    • + +
  • HADOOP-7685. + Major bug reported by devaraj.k and fixed by devaraj.k (scripts)
    + Issues with hadoop-common-project\hadoop-common\src\main\packages\hadoop-setup-conf.sh file
    +
    hadoop-common-project\hadoop-common\src\main\packages\hadoop-setup-conf.sh has following issues
    1. check_permission does not work as expected if there are two folders with $NAME as part of their name inside $PARENT
    e.g. /home/hadoop/conf, /home/hadoop/someconf,
    The result of `ls -ln $PARENT | grep -w $NAME| awk '{print $3}'` is non zero..it is 0 0 and hence the following if check becomes true.
    {code:xml}
    if [ "$OWNER" != "0" ]; then
    RESULT=1
    break
    fi
    {code}

    2. Spelling mistake
    {code:xml}
    H...
    • + +
  • HADOOP-7691. + Major bug reported by gkesavan and fixed by eyang
    + hadoop deb pkg should take a diff group id
    +
    ubuntu - 11.04 is using group id 114 for gdm.
    hadoop deb pkg should pickup a different groupid.
    • +
  • HDFS-142. Blocker bug reported by rangadi and fixed by dhruba
    In 0.20, move blocks being written into a blocksBeingWritten directory
    @@ -395,7 +421,7 @@
  • HDFS-2318. Major sub-task reported by szetszwo and fixed by szetszwo
    Provide authentication to webhdfs using SPNEGO
    -
    • +
    Added two new conf properties dfs.web.authentication.kerberos.principal and dfs.web.authentication.kerberos.keytab for the SPNEGO servlet filter.


  • HDFS-2320. Major bug reported by sureshms and fixed by sureshms (data-node, hdfs client, name-node)
    @@ -425,7 +451,7 @@
  • HDFS-2338. Major sub-task reported by jnp and fixed by jnp
    Configuration option to enable/disable webhdfs.
    -
    We should add a configuration option to enable/disable webhdfs.
    • +
    Added a conf property dfs.webhdfs.enabled for enabling/disabling webhdfs.


  • HDFS-2340. Major sub-task reported by szetszwo and fixed by szetszwo
    @@ -450,13 +476,23 @@
  • HDFS-2361. Critical bug reported by rajsaha and fixed by jnp (name-node)
    hftp is broken
    -
    Distcp with hftp is failing.


    $hadoop distcp hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp 1316814737/as
    11/09/23 21:52:33 INFO tools.DistCp: srcPaths=[hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp]
    11/09/23 21:52:33 INFO tools.DistCp: destPath=1316814737/as
    Retrieving token from: https://<NN IP>:50470/getDelegationToken
    Retrieving token from: https://<NN IP>:50470/getDelegationToken?renewer=mapred
    11/09/23 21:52:34 INFO security.TokenCache: Got dt for hftp://<NNh...
    • +
    Distcp with hftp is failing.

    {noformat}
    $hadoop distcp hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp 1316814737/as
    11/09/23 21:52:33 INFO tools.DistCp: srcPaths=[hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp]
    11/09/23 21:52:33 INFO tools.DistCp: destPath=1316814737/as
    Retrieving token from: https://<NN IP>:50470/getDelegationToken
    Retrieving token from: https://<NN IP>:50470/getDelegationToken?renewer=mapred
    11/09/23 21:52:34 INFO security.TokenCache: Got dt for h...
  • HDFS-2366. Major bug reported by arpitgupta and fixed by szetszwo
    webhdfs throws a npe when ugi is null from getDelegationToken
    • +
  • HDFS-2368. + Major bug reported by arpitgupta and fixed by szetszwo
    + defaults created for web keytab and principal, these properties should not have defaults
    +
    the following defaults are set in hdfs-defaults.xml

    <property>
    <name>dfs.web.authentication.kerberos.principal</name>
    <value>HTTP/${dfs.web.hostname}@${kerberos.realm}</value>
    <description>
    The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.

    The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos
    HTTP SPENGO specification.
    </description>
    </property>

    <property>
    <name>dfs.web.authentication.kerberos.keytab</name>
    <value>${user.home}/dfs.web....
    • + +
  • HDFS-2373. + Major bug reported by arpitgupta and fixed by arpitgupta
    + Commands using webhdfs and hftp print unnecessary debug information on the console with security enabled
    +
    run an hdfs command using either hftp or webhdfs and it prints the following line to the console (system out)

    Retrieving token from: https://NN_HOST:50470/getDelegationToken


    Probably in the code where we get the delegation token. This should be removed as people using the dfs commands to get a handle to the content such as dfs -cat will now get an extra line that is not part of the actual content. This should either be only in the log or not logged at all.
    • +
  • HDFS-2375. Blocker bug reported by sureshms and fixed by sureshms (hdfs client)
    TestFileAppend4 fails in 0.20.205 branch
    @@ -557,6 +593,16 @@ TestSleepJob fails
    TestSleepJob fails, it was intended to be used in other tests for MAPREDUCE-2981.
    • +
  • MAPREDUCE-3081. + Major bug reported by vitthal_gogate and fixed by (contrib/vaidya)
    + Change the name format for hadoop core and vaidya jar to be hadoop-{core/vaidya}-{version}.jar in vaidya.sh
    +
    contrib/vaidya/bin/vaidya.sh script fixed to use appropriate jars and classpath


    • + +
  • MAPREDUCE-3112. + Major bug reported by eyang and fixed by eyang (contrib/streaming)
    + Calling hadoop cli inside mapreduce job leads to errors
    +
    Removed inheritance of certain server environment variables (HADOOP_OPTS and HADOOP_ROOT_LOGGER) in task attempt process. <br/>


    • +