hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siyao Meng (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-14092) Remove two-step create/append in WebHdfsFileSystem
Date Wed, 28 Nov 2018 00:51:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701179#comment-16701179
] 

Siyao Meng edited comment on HDFS-14092 at 11/28/18 12:50 AM:
--------------------------------------------------------------

I've taken a look at the client.
{code:java}
  /** Expects HTTP response 307 "Temporary Redirect". */
  public static class TemporaryRedirectOp implements Op {
    static final TemporaryRedirectOp CREATE = new TemporaryRedirectOp(
        PutOpParam.Op.CREATE);
    static final TemporaryRedirectOp APPEND = new TemporaryRedirectOp(
        PostOpParam.Op.APPEND);
    static final TemporaryRedirectOp OPEN = new TemporaryRedirectOp(
        GetOpParam.Op.OPEN);
    static final TemporaryRedirectOp GETFILECHECKSUM = new TemporaryRedirectOp(
        GetOpParam.Op.GETFILECHECKSUM);
...
{code}
Only for those 4 operations the WebHDFS client would: 1. Send a request to server and expect
307 Temporary Redirect; 2. Grab the new URL from HTTP header Location key and send a second
request to it (which points to a DN). WebHdfsFileSystem#connect(URL) performs the logic.
-So my understanding for the problem is that, the Java 6 library was faulty so it couldn't
redirect the request automatically (like user browsers normally do when encountered HTTP 3xx).
Therefore, we had to do it manually. CMIIW-
I just read HDFS-2540 which added this two-step code. And I am a bit confused. What is the
expected logic if we want to remove two-step?
Any comments?
CC [~szetszwo]


was (Author: smeng):
I've taken a look at the client.
{code:java}
  /** Expects HTTP response 307 "Temporary Redirect". */
  public static class TemporaryRedirectOp implements Op {
    static final TemporaryRedirectOp CREATE = new TemporaryRedirectOp(
        PutOpParam.Op.CREATE);
    static final TemporaryRedirectOp APPEND = new TemporaryRedirectOp(
        PostOpParam.Op.APPEND);
    static final TemporaryRedirectOp OPEN = new TemporaryRedirectOp(
        GetOpParam.Op.OPEN);
    static final TemporaryRedirectOp GETFILECHECKSUM = new TemporaryRedirectOp(
        GetOpParam.Op.GETFILECHECKSUM);
...
{code}
Only for those 4 operations the WebHDFS client would: 1. Send a request to server and expect
307 Temporary Redirect; 2. Grab the new URL from HTTP header Location key and send a second
request to it (which points to a DN). WebHdfsFileSystem#connect(URL) performs the logic.

 

So my understanding for the problem is that, the Java 6 library was faulty so it couldn't
redirect the request automatically (like user browsers normally do when encountered HTTP 3xx).
Therefore, we had to do it manually. CMIIW

> Remove two-step create/append in WebHdfsFileSystem
> --------------------------------------------------
>
>                 Key: HDFS-14092
>                 URL: https://issues.apache.org/jira/browse/HDFS-14092
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>    Affects Versions: 3.2.0
>            Reporter: Daniel Templeton
>            Assignee: Siyao Meng
>            Priority: Major
>
> Per the javadoc on the {{WebHdfsFileSystem.connect()}} method:
> {code}    /**
>      * Two-step requests redirected to a DN
>      *
>      * Create/Append:
>      * Step 1) Submit a Http request with neither auto-redirect nor data.
>      * Step 2) Submit another Http request with the URL from the Location header
>      * with data.
>      *
>      * The reason of having two-step create/append is for preventing clients to
>      * send out the data before the redirect. This issue is addressed by the
>      * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3.
>      * Unfortunately, there are software library bugs (e.g. Jetty 6 http server
>      * and Java 6 http client), which do not correctly implement "Expect:
>      * 100-continue". The two-step create/append is a temporary workaround for
>      * the software library bugs.
>      *
>      * Open/Checksum
>      * Also implements two-step connects for other operations redirected to
>      * a DN such as open and checksum
>      */{code}
> We should validate that it's safe to remove the two-step process and do so.  FYI, [~smeng].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message