hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
Date Wed, 01 Aug 2018 01:00:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564576#comment-16564576
] 

Chen Liang edited comment on HDFS-13767 at 8/1/18 12:59 AM:
------------------------------------------------------------

Post WIP.v004 patch. The main changes are:
 # changed to {{getCorrectLastAppliedOrWrittenTxId}} as Erik suggested.
 # Add a simple unit test and had some minor change to {{TestObserverNode}}.

I was encountering some issues when running test, here are some suspects I had on TestObserverNode:

Currently with msync, every single call to Observer needs to catch up state id. While when {{ObserverReadProxyProvider}}
is created, it makes {{reportBadBlocks}} and {{checkAccess}} calls. These calls also need
to catch up state id. This means every time when  {{ObserverReadProxyProvider}} is created
there will be a wait... And since every unit test creates a dfs cluster, for every single
unit test in {{TestObserverNode}}, there will be such a wait. I could set the period to a
small time, but that introduces other issues, because several tests depend on explicitly
made edits tailing calls. So I set the periods to a smaller, but not too small number...and
tests still quite some time to run.

I think the right fix would be to have {{ObserverReadProxyProvider}} not make those calls
on being created. But make a new special call that bypasses the catching up (if possible).
This is one of those hacky code we are planing to optimize in the future. And also, the
ongoing work HDFS-13523 may also simplify the tests. For now, I tend to believe having these
slow tests may be fine.


was (Author: vagarychen):
Post WIP.v004 patch. The main changes are:
 # changed to {{getCorrectLastAppliedOrWrittenTxId}} as Erik suggested.
 # Add a simple unit test and had some minor change to {{TestObserverNode}}.

I was encountering some issues when running test, here are some suspects I had on TestObserverNode:

Currently with msync, every single call to Observer needs to catch up state id. While when {{ObserverReadProxyProvider}}
is created, it makes {{reportBadBlocks}} and {{checkAccess}} calls. These calls are also read,
so they also need to catch up state id. This means every time when  {{ObserverReadProxyProvider}} is
created there will be a wait... And since every unit test creates a dfs cluster, for every
single unit test in {{TestObserverNode}}, there will be such a wait. I could set the period to
a small time, but that introduces other issues, because several tests depend on explicitly
made edits tailing calls. So I set the periods to a smaller, but not too small number...and
tests still quite some time to run.

I think the right fix would be to have {{ObserverReadProxyProvider}} not make those read
calls on being created. This is one of those hacky code we are planing to optimize in the
future. And also, the ongoing work HDFS-13523 may also simplify the tests. For now, I tend
to believe having these slow tests may be fine.

> Add msync server implementation.
> --------------------------------
>
>                 Key: HDFS-13767
>                 URL: https://issues.apache.org/jira/browse/HDFS-13767
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Chen Liang
>            Assignee: Chen Liang
>            Priority: Major
>         Attachments: HDFS-13767.WIP.001.patch, HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch,
HDFS-13767.WIP.004.patch
>
>
> This is a followup on HDFS-13688, where msync API is introduced to {{ClientProtocol}}
but the server side implementation is missing. This is Jira is to implement the server side
logic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message