hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11644) DFSStripedOutputStream should not implement Syncable
Date Thu, 13 Apr 2017 09:04:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967317#comment-15967317

Steve Loughran commented on HDFS-11644:

I've slowly started on output streams, HADOOP-13327, [doc here|https://github.com/steveloughran/hadoop/blob/s3/HADOOP-13327-outputstream-spec/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md]

No tests, etc.

In that draft I've said, "The purpose of Syncable interface is to provide guarantees that
data is written to a filesystem for both visibility and durability."

I can see the benefit of runtime checks. However, I don't want a whole new set of methods
for features, as they become a problem too: probe for interface, cast, query, maintain.  Like
you say, at least with Java 8 you can add new methods: I was about to say "no can do" for
a new method there.

Runtime probes allow for dynamic behaviour based on FS semantics, things we need for stuff
like "case-sensitive-fs", "create-consistency", etc: things which are a function of the specific
FS instance, rather than just the {{org.apache.hadoop.FileSystem}} subclasses.

I'd advocate what we've been discussing in HADOOP-9565, some single method {{boolean hasFeature(String)}},
where a feature can be probed for. Implement that and have one probe "syncable" to make: base
classes can implement, subclasses can override that specific string. Add that method as a
new Interface and we can adopt it across all our streams.

> DFSStripedOutputStream should not implement Syncable
> ----------------------------------------------------
>                 Key: HDFS-11644
>                 URL: https://issues.apache.org/jira/browse/HDFS-11644
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
>            Assignee: Manoj Govindassamy
>              Labels: hdfs-ec-3.0-must-do
> FSDataOutputStream#hsync checks if a stream implements Syncable, and if so, calls hsync.
Otherwise, it just calls flush. This is used, for instance, by YARN's FileSystemTimelineWriter.
> DFSStripedOutputStream extends DFSOutputStream, which implements Syncable. However, DFSStripedOS
throws a runtime exception when the Syncable methods are called.
> We should refactor the inheritance structure so DFSStripedOS does not implement Syncable.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message