Return-Path: <hdfs-issues-return-177181-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 7A0B0200BEF
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Wed,  4 Jan 2017 23:17:05 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 78EC5160B3A; Wed,  4 Jan 2017 22:17:05 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id C39E8160B21
	for <archive-asf-public@cust-asf.ponee.io>; Wed,  4 Jan 2017 23:17:04 +0100 (CET)
Received: (qmail 66311 invoked by uid 500); 4 Jan 2017 22:16:58 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 66155 invoked by uid 99); 4 Jan 2017 22:16:58 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2017 22:16:58 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id AAEC42C2A6D
	for <hdfs-issues@hadoop.apache.org>; Wed,  4 Jan 2017 22:16:58 +0000 (UTC)
Date: Wed, 4 Jan 2017 22:16:58 +0000 (UTC)
From: "Jing Zhao (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13030235.1482441744000.643690.1483568218697@Atlassian.JIRA>
In-Reply-To: <JIRA.13030235.1482441744000@Atlassian.JIRA>
References: <JIRA.13030235.1482441744000@Atlassian.JIRA> <JIRA.13030235.1482441744376@arcas>
Subject: [jira] [Commented] (HDFS-11273) Move TransferFsImage#doGetUrl
 function to a Util class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Wed, 04 Jan 2017 22:17:05 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15799513#comment-15799513 ] 

Jing Zhao commented on HDFS-11273:
----------------------------------

Thanks for updating the patch, [~hkoneru]! The updated patch is almost there. I have several extra minor comments (sorry I did not mention them in my last review...):
# The new Util#setTimeout method may no longer only load timeout value from "DFS_IMAGE_TRANSFER_TIMEOUT_KEY". Thus the code loading timeout from configuration can be left in TransferFsImage#doGetUrl.
{code}
+  /**
+   * Sets a timeout value in millisecods for the Http connection.
+   * @param connection the Http connection for which timeout needs to be set
+   * @param timeout value to be set as timeout in milliseconds
+   */
+  public static void setTimeout(HttpURLConnection connection, int timeout) {
+    if (timeout <= 0) {
+      Configuration conf = new HdfsConfiguration();
+      timeout = conf.getInt(
+          DFSConfigKeys.DFS_IMAGE_TRANSFER_TIMEOUT_KEY,
+          DFSConfigKeys.DFS_IMAGE_TRANSFER_TIMEOUT_DEFAULT);
+      LOG.info("Image Transfer timeout configured to " + timeout
+          + " milliseconds");
+    }
+
+    if (timeout > 0) {
+      connection.setConnectTimeout(timeout);
+      connection.setReadTimeout(timeout);
+    }
+  }
{code}
# HttpGetFailedException can be defined as an upper level class and be moved to the o.a.h.hdfs.server.common package.
# The following code can be reformatted.
{code}
+  public static MD5Hash doGetUrl(URL url, List<File> localPaths,
+      Storage dstStorage, boolean getChecksum, URLConnectionFactory
+      connectionFactory, int ioFileBufferSize, boolean isSpnegoEnabled, int
+      timeout) throws IOException {
{code}

> Move TransferFsImage#doGetUrl function to a Util class
> ------------------------------------------------------
>
>                 Key: HDFS-11273
>                 URL: https://issues.apache.org/jira/browse/HDFS-11273
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Hanisha Koneru
>            Assignee: Hanisha Koneru
>         Attachments: HDFS-11273.000.patch, HDFS-11273.001.patch
>
>
> TransferFsImage#doGetUrl downloads files from the specified url and stores them in the specified storage location. HDFS-4025 plans to synchronize the log segments in JournalNodes. If a log segment is missing from a JN, the JN downloads it from another JN which has the required log segment. We need TransferFsImage#doGetUrl and TransferFsImage#receiveFile to accomplish this. 
> So we propose to move the said functions to a Utility class so as to be able to use it for JournalNode syncing as well, without duplication of code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org