hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
Date Wed, 05 Dec 2012 18:53:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510680#comment-13510680

Yu Li commented on HDFS-4262:

Thanks Alejandro.

After applying the two patches you supplied, UT result in my env is like:
Tests in error:
  testOperation[0](org.apache.hadoop.fs.http.client.TestWebhdfsFileSystem): java.io.IOException:
Server returned HTTP response code: 500 for URL: http://bdvm072.svl.ibm.com:48763/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=biadmin
  testOperationDoAs[0](org.apache.hadoop.fs.http.client.TestWebhdfsFileSystem): java.io.IOException:
Server returned HTTP response code: 401 for URL: http://bdvm072.svl.ibm.com:57519/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=user1
  testOperation[0](org.apache.hadoop.fs.http.client.TestHttpFSFileSystem): java.io.IOException:
Server returned HTTP response code: 500 for URL: http://bdvm072.svl.ibm.com:57757/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=biadmin
  testOperationDoAs[0](org.apache.hadoop.fs.http.client.TestHttpFSFileSystem): java.io.IOException:
Server returned HTTP response code: 401 for URL: http://bdvm072.svl.ibm.com:56289/webhdfs/v1/?op=GETHOMEDIRECTORY&doas=user1

Tests run: 177, Failures: 0, Errors: 4, Skipped: 0

Then I did some investigation and made another patch, "03-resolve-proxyuser-related-issue.patch",
which could resolve the UT failure. I also merged the three patches together to "HDFS-4262-github.patch",
as attached.

After getting all UT pass, I also build-out a httpfs tar ball and tested it on a hadoop-1.0.3
environment, and most function worked. However, I found the API documents, either the ones
attached with HDFS-2178 or on http://cloudera.github.com/httpfs/UsingHttpTools.html, are out
of date. Like to get the name dir of one specified user, the request should be:
<p><i>curl -X GET "http://shihc024-public.cn.ibm.com:14000/webhdfs/v1?user.name=biadmin&op=gethomedirectory"</i></p>
rather than
<p><i>curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=babu&op=homedir"</i></p>

Anybody could tell me where I can get the latest API doc, so I could apply a full sanity test?
> Backport HTTPFS to Branch 1
> ---------------------------
>                 Key: HDFS-4262
>                 URL: https://issues.apache.org/jira/browse/HDFS-4262
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>         Environment: IBM JDK, RHEL 6.3
>            Reporter: Eric Yang
>            Assignee: Yu Li
>         Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch,
03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch
> There are interests to backport HTTPFS back to Hadoop 1 branch.  After the initial investigation,
there're quite some changes in HDFS-2178, and several related patches, including:
> HDFS-2284 Write Http access to HDFS
> HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings
> HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs
> HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk
> HDFS-2658 HttpFS introduced 70 javadoc warnings
> The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X,
which  code base has been refactored a lot and quite different from 1.X, so it seems we have
to backport the changes manually.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message