hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akira Ajisaka (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-13719) Docs around dfs.image.transfer.timeout are misleading
Date Tue, 30 Oct 2018 01:20:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Akira Ajisaka updated HDFS-13719:
---------------------------------
    Component/s: documentation

> Docs around dfs.image.transfer.timeout are misleading
> -----------------------------------------------------
>
>                 Key: HDFS-13719
>                 URL: https://issues.apache.org/jira/browse/HDFS-13719
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 3.1.0
>            Reporter: Kitti Nanasi
>            Assignee: Kitti Nanasi
>            Priority: Major
>              Labels: hdfs
>             Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.4
>
>         Attachments: HDFS-13719-branch-2.001.patch, HDFS-13719.001.patch, HDFS-13719.002.patch
>
>
> The Jira https://issues.apache.org/jira/browse/HDFS-1490 added the parameter dfs.image.transfer.timeout
to HDFS. From the patch (and checking the current code), we can see this parameter governs
a socket timeout on the a java.net.HttpURLConnection object:
> {code:java}
> +    if (timeout <= 0) {
> +      // Set the ping interval as timeout
> +      Configuration conf = new HdfsConfiguration();
> +      timeout = conf.getInt(DFSConfigKeys.DFS_IMAGE_TRANSFER_TIMEOUT_KEY,
> +          DFSConfigKeys.DFS_IMAGE_TRANSFER_TIMEOUT_DEFAULT);
> +    }
> +
> +    if (timeout > 0) {
> +      connection.setConnectTimeout(timeout);
> +      connection.setReadTimeout(timeout);
> +    }
> +
> {code}
> In the above 'connection' is a java.net.HttpURLConnection.
> There is a general disbelief in the community that dfs.image.transfer.timeout is the
time the entire image must transfer within, however that does not appear to be the case. The
timeout is actually the max time the client will block on the socket before giving up if it
cannot get data to read. I guess the idea here is to protect the client from hanging forever
if the server hangs.
> The docs in hdfs-site.xml are partly what causes this confusion, as they are very misleading:
> {code:xml}
> <property>
>   <name>dfs.image.transfer.timeout</name>
>   <value>60000</value>
>   <description>
>         Socket timeout for image transfer in milliseconds. This timeout and the related
>         dfs.image.transfer.bandwidthPerSec parameter should be configured such
>         that normal image transfer can complete successfully.
>         This timeout prevents client hangs when the sender fails during
>         image transfer. This is socket timeout during image tranfer.
>   </description>
> </property>
> {code}
> The start and end of the statement is accurate, but the part "This timeout and the related
dfs.image.transfer.bandwidthPerSec parameter should be configured such that normal image transfer
can complete successfully." is misleading. There is almost never a reason to change the above
in conjunction with the bandwidth setting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message