hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5946) Webhdfs DN choosing code is flawed
Date Thu, 13 Feb 2014 20:01:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900671#comment-13900671

Jing Zhao commented on HDFS-5946:

But for reading through webhdfs you also want to get the data locality. In that case the client
wants the local DN or the closest DN for reading. Currently the sorting logic takes this part
into account and the first DN in the list is the best one. That's also why in the original/current
implementation only web UI uses random pick.

> Webhdfs DN choosing code is flawed
> ----------------------------------
>                 Key: HDFS-5946
>                 URL: https://issues.apache.org/jira/browse/HDFS-5946
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, webhdfs
>    Affects Versions: 3.0.0, 2.4.0
>            Reporter: Daryn Sharp
>            Priority: Critical
> HDFS-5891 improved the performance of redirecting webhdfs clients to a DN.  Instead of
attempting a connection with a 1-minute timeout, the NN skips decommissioned nodes.
> The logic appears flawed.  It finds the index of the first decommissioned node, if any,
> * Throws an exception if index = 0, even if other nodes later in the list are not decommissioned.
> * Else picks a random node prior to the index.  Let's say there are 10 replicas, 2nd
location is decommissioned.  All clients will be redirected to the first location even though
there are 8 other valid locations.

This message was sent by Atlassian JIRA

View raw message