hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
Date Wed, 05 Dec 2012 18:53:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510680#comment-13510680
] 

Yu Li commented on HDFS-4262:
-----------------------------

Thanks Alejandro.

After applying the two patches you supplied, UT result in my env is like:
===============================================================
Tests in error:
  testOperation[0](org.apache.hadoop.fs.http.client.TestWebhdfsFileSystem): java.io.IOException:
Server returned HTTP response code: 500 for URL: http://bdvm072.svl.ibm.com:48763/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=biadmin
  testOperationDoAs[0](org.apache.hadoop.fs.http.client.TestWebhdfsFileSystem): java.io.IOException:
Server returned HTTP response code: 401 for URL: http://bdvm072.svl.ibm.com:57519/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=user1
  testOperation[0](org.apache.hadoop.fs.http.client.TestHttpFSFileSystem): java.io.IOException:
Server returned HTTP response code: 500 for URL: http://bdvm072.svl.ibm.com:57757/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=biadmin
  testOperationDoAs[0](org.apache.hadoop.fs.http.client.TestHttpFSFileSystem): java.io.IOException:
Server returned HTTP response code: 401 for URL: http://bdvm072.svl.ibm.com:56289/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=user1

Tests run: 177, Failures: 0, Errors: 4, Skipped: 0
===============================================================

Then I did some investigation and made another patch, "03-resolve-proxyuser-related-issue.patch",
which could resolve the UT failure. I also merged the three patches together to "HDFS-4262-github.patch",
as attached.

After getting all UT pass, I also build-out a httpfs tar ball and tested it on a hadoop-1.0.3
environment, and most function worked. However, I found the API documents, either the ones
attached with HDFS-2178 or on http://cloudera.github.com/httpfs/UsingHttpTools.html, are out
of date. Like to get the name dir of one specified user, the request should be:
<p><i>curl -X GET "http://shihc024-public.cn.ibm.com:14000/webhdfs/v1?user.name=biadmin&op=gethomedirectory"</i></p>
rather than
<p><i>curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=babu&op=homedir"</i></p>

Anybody could tell me where I can get the latest API doc, so I could apply a full sanity test?
                
> Backport HTTPFS to Branch 1
> ---------------------------
>
>                 Key: HDFS-4262
>                 URL: https://issues.apache.org/jira/browse/HDFS-4262
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>         Environment: IBM JDK, RHEL 6.3
>            Reporter: Eric Yang
>            Assignee: Yu Li
>         Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch,
03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch
>
>
> There are interests to backport HTTPFS back to Hadoop 1 branch.  After the initial investigation,
there're quite some changes in HDFS-2178, and several related patches, including:
> HDFS-2284 Write Http access to HDFS
> HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings
> HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs
> HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk
> HDFS-2658 HttpFS introduced 70 javadoc warnings
> The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X,
which  code base has been refactored a lot and quite different from 1.X, so it seems we have
to backport the changes manually.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message