hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9020) Support hflush/hsync in WebHDFS
Date Fri, 04 Sep 2015 00:09:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730075#comment-14730075

Chris Douglas commented on HDFS-9020:

bq. caching dfsclients is nontrivial as the cache uses the instances of UGI, but not the principals
of it
Are you referring to the FileSystem cache? I haven't traced this, but the idea was to map
an incoming {{POST}} to the HDFS stream, not just the client instance. But I may not catch
your meaning... do you have a pointer into the code?

bq. the client cannot recover once the stateful DN is down.
[~daryn] pointed out that we need a timeout for idle connections, which should let us time
out the connection and recover, at least to the extent we can in the existing protocol. The
existing implementation already relies on TCP timeouts... and is intolerant of multiple failures,
particularly for the append case, right? I'm not dismissing the complexity of handling state,
but is it adding novel failure modes?

The HTTP/2 work (HDFS-7966) should dominate WebSockets, which is pretty raw. The goal is shared:
breaking up the stream into a sequence of {{\[PUT\]POST\*}} operations is trying to be message-based.
Similar to the point Todd and Stack [made|https://issues.apache.org/jira/browse/HDFS-7966?focusedCommentId=14588913&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14588913],
a WebSockets protocol is yet-another thing to maintain...

> Support hflush/hsync in WebHDFS
> -------------------------------
>                 Key: HDFS-9020
>                 URL: https://issues.apache.org/jira/browse/HDFS-9020
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>            Reporter: Chris Douglas
>         Attachments: HDFS-9020-alt.txt
> In the current implementation, hflush/hsync have no effect on WebHDFS streams, particularly
w.r.t. visibility to other clients. This proposes to extend the protocol and implementation
to enable this functionality.

This message was sent by Atlassian JIRA

View raw message