hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
Date Wed, 26 Oct 2016 15:39:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608805#comment-15608805
] 

Sean Mackrory edited comment on HDFS-10702 at 10/26/16 3:39 PM:
----------------------------------------------------------------

I think we should also consider ways to not send requests to multiple SNNs. I did some testing,
with the latest patch and while this does indeed decrease traffic to the NN and increase performance
when a single SNN is there, if you have multiple SNNs this only decreases traffic to the NN
- it actually also decreases the performance of the client and would further increase overall
network usage. I don't think we need to be particularly sophisticated about picking the most
ideal SNN, especially since this feature inherently accepts the idea of not having the most
up to date metadata.


was (Author: mackrorysd):
I think we should also consider ways to not send requests to multiple SNNs. I did some testing,
and while this does indeed decrease traffic to the NN and increase performance when a single
SNN is there, if you have multiple SNNs this only decreases traffic to the NN - it actually
also decreases the performance of the client and would further increase overall network usage.
I don't think we need to be particularly sophisticated about picking the most ideal SNN, especially
since this feature inherently accepts the idea of not having the most up to date metadata.

> Add a Client API and Proxy Provider to enable stale read from Standby
> ---------------------------------------------------------------------
>
>                 Key: HDFS-10702
>                 URL: https://issues.apache.org/jira/browse/HDFS-10702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Jiayi Zhou
>            Assignee: Jiayi Zhou
>            Priority: Minor
>         Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, HDFS-10702.003.patch,
HDFS-10702.004.patch, HDFS-10702.005.patch, StaleReadfromStandbyNN.pdf
>
>
> Currently, clients must always talk to the active NameNode when performing any metadata
operation, which means active NameNode could be a bottleneck for scalability. One way to solve
this problem is to send read-only operations to Standby NameNode. The disadvantage is that
it might be a stale read. 
> Here, I'm thinking of adding a Client API to enable/disable stale read from Standby which
gives Client the power to set the staleness restriction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message