hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12943) Consistent Reads from Standby Node
Date Sun, 16 Dec 2018 19:36:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722550#comment-16722550

Brahma Reddy Battula commented on HDFS-12943:

{quote}I think when we discuss a "request", we need to differentiate an RPC request originating
from a Java application (MapReduce task, etc.) vs. a CLI request. The former will be the vast
majority of operations on a typical cluster, so I would argue that optimizing for the performance
and efficiency of that usage is much more important.
Agree, I Could have mentioned CLI. But getHAServiceState() call from ORP which taken 2s+ as
I mentioned above.Bytheway My intent was when read/write are combined in single application
how much will be impact as it needs switch?

Just for curiosity,,do we've write benchmarks with and without ORP,as I didn't find from
HDFS-14058 and HDFS-14059?
{quote}1.Are you running with HDFS-13873? With this patch (only committed yesterday so I doubt
you have it) the exception thrown should be more meaningful.
Yes,with latest HDFS-12943 branch.
{quote}2.Did you remember to enable in-progress edit log tailing?
Yes,Enabled for three NN's
{quote}3.Was this run on an almost completely stagnant cluster (no other writes)? This can
make the ANN flush its edits to the JNs less frequently, increasing the lag time between ANN
and Observer.
Yes,no other writes.

 Tried the following test with and with ORF,Came to know it's(perf impact) based on the tailing
edits("*dfs.ha.tail-edits.period") which is default 1m.(In tests, it's 100MS)..*
 public void testSimpleRead() throws Exception {
 long avg=0;
 long avgL=0;
 long avgC=0;
 int num = 100;
 for (int i = 0; i < num; i++) {
 Path testPath1 = new Path(testPath, "test1"+i);
 long startTime=System.currentTimeMillis();
 assertTrue(dfs.mkdirs(testPath1, FsPermission.getDefault()));
 long l = System.currentTimeMillis() - startTime;
 System.out.println("time TakenL1: "+i+" : "+l);
 avg = avg+l;
 long startTime2=System.currentTimeMillis();
 long C = System.currentTimeMillis() - startTime2;
 System.out.println("time TakengetContentSummary: "+i+" : "+ C);
 avgC = avgC+C;
 long startTime1=System.currentTimeMillis();
 long L = System.currentTimeMillis() - startTime1;
 System.out.println("time TakengetFileStatus: "+i+" : "+ L);
 avgL = avgL+L;
 System.out.println("AVG: mkDir: "+avg/num+" List: "+avgL/num+" Cont: "+avgC/num);
Apart from the perf i have following queries.
 i) Did we try with C/CPP client..?
 ii)are we planning separate metrics for observer reads(Client Side),Application like mapred
might helpful for  job counters?


> Consistent Reads from Standby Node
> ----------------------------------
>                 Key: HDFS-12943
>                 URL: https://issues.apache.org/jira/browse/HDFS-12943
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs
>            Reporter: Konstantin Shvachko
>            Priority: Major
>         Attachments: ConsistentReadsFromStandbyNode.pdf, ConsistentReadsFromStandbyNode.pdf,
HDFS-12943-001.patch, HDFS-12943-002.patch, TestPlan-ConsistentReadsFromStandbyNode.pdf
> StandbyNode in HDFS is a replica of the active NameNode. The states of the NameNodes
are coordinated via the journal. It is natural to consider StandbyNode as a read-only replica.
As with any replicated distributed system the problem of stale reads should be resolved. Our
main goal is to provide reads from standby in a consistent way in order to enable a wide range
of existing applications running on top of HDFS.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message