Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 3 Dec 2013 18:42:36 +0000 (UTC)
From: "Haohui Mai (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12554862.1336711540226.63944.1386096156491@arcas>
In-Reply-To: <JIRA.12554862.1336711540226@arcas>
References: <JIRA.12554862.1336711540226@arcas>
Subject: [jira] [Commented] (HDFS-3405) Checkpointing should use HTTP POST
 or PUT instead of GET-GET to send merged fsimages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13838014#comment-13838014 ] 

Haohui Mai commented on HDFS-3405:
----------------------------------

I'll take a more detailed look this week. After a quick skim, it seems that you're using Apache http client to make the put request -- this won't work if it is going through HTTPS channels, since it does not load the certificates. The recommended way is to open the connection through URLConnectionFactory and to change the connConfigurator. Here is the sketch:

{code}
class PutConnConfigurator implements ConnectionConfigurator {
  private final ConnectionConfigurator prev;
  private HttpURLConnection configure(HttpURLConnection conn) {
    prev.configure(prev);
    conn.setRequestMethod("PUT");
    return conn;
  }
  PutConnConfigurator(...) { }
}

ConnConfigurator putConf = new PutConnConfigurator(factory.getConnConfigurator());
URLConnectionFactory newFactory = new URLConnectionFactory(putConf);
URLConnection conn = newFactory.openConnection(...);
{code}

> Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged fsimages
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-3405
>                 URL: https://issues.apache.org/jira/browse/HDFS-3405
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 1.0.0, 3.0.0, 2.0.5-alpha
>            Reporter: Aaron T. Myers
>            Assignee: Vinay
>         Attachments: HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch
>
>
> As Todd points out in [this comment|https://issues.apache.org/jira/browse/HDFS-3404?focusedCommentId=13272986&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13272986], the current scheme for a checkpointing daemon to upload a merged fsimage file to an NN is to issue an HTTP get request to tell the target NN to issue another GET request back to the checkpointing daemon to retrieve the merged fsimage file. There's no fundamental reason the checkpointing daemon can't just use an HTTP POST or PUT to send back the merged fsimage file, rather than the double-GET scheme.


--
This message was sent by Atlassian JIRA
(v6.1#6144)