hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Plamen Jeliazkov (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
Date Fri, 03 Aug 2018 19:58:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568674#comment-16568674
] 

Plamen Jeliazkov edited comment on HDFS-13767 at 8/3/18 7:57 PM:
-----------------------------------------------------------------

Hey Chen,

Change looks pretty good so far; couple things.

(1) I like the idea of splitting AlignmentContext into client and server interfaces. I think
originally Konstantin wanted just a single interface but I think its clear there is a difference.
I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow
the server-side version of AlignmentContext to better handle defering / re-queue'ing calls.

(2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start
and the assert:
{code:java}
assertFalse(readSucceed.get());
{code}
However I found that the test would fail if I did. Which makes me question the unit test.
I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but
seems it is updating anyway.
I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway.
Please let me know if I am missing something about this test; I will look further.

(3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we rename
it?

(4) [~xkrogen], [~shv], and others -- I was curious about making msync pause at the server
level. What if we had msync hang at the client side instead? This way we aren't impacting
Observer queues with many deferred calls. We could simply have the client check up with the
Observers' states and so rather than re-queue'ing we throw a StandbyException (or something)
and force the client to maybe pause and retry the msync call on the same or a different Observer.
If the msync call succeeds then we know we can use the particular Observer that it succeeded
on. Thoughts?




was (Author: zero45):
Hey Chen,

Change looks pretty good so far; couple things.

(1) I like the idea of splitting AlignmentContext into client and server interfaces. I think
originally Konstantin wanted just a single interface but I think its clear there is a difference.
I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow
the server-side version of AlignmentContext to better handle defering / re-queue'ing calls.

(2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start
and the assert:
{code:java}
assertFalse(readSucceed.get());
{code}
However I found that the test would fail if I did. Which makes me question the unit test.
I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but
seems it is updating anyway.
I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway.
Please let me know if I am missing something about this test; I will look further.

(3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we rename
it?

(4) [~xkrogen], [~shv], and others -- I was curios about making msync pause at the server
level. What if we had msync hang at the client side instead? This way we aren't impacting
Observer queues with many deferred calls. We could simply have the client check up with the
Observers' states and so rather than re-queue'ing we throw a StandbyException (or something)
and force the client to maybe pause and retry the msync call on the same or a different Observer.
If the msync call succeeds then we know we can use the particular Observer that it succeeded
on. Thoughts?



> Add msync server implementation.
> --------------------------------
>
>                 Key: HDFS-13767
>                 URL: https://issues.apache.org/jira/browse/HDFS-13767
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Chen Liang
>            Assignee: Chen Liang
>            Priority: Major
>         Attachments: HDFS-13767-HDFS-12943.001.patch, HDFS-13767-HDFS-12943.002.patch,
HDFS-13767.WIP.001.patch, HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch
>
>
> This is a followup on HDFS-13688, where msync API is introduced to {{ClientProtocol}}
but the server side implementation is missing. This is Jira is to implement the server side
logic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message