hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
Date Mon, 01 May 2017 18:16:04 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-13327:
------------------------------------
    Attachment: HADOOP-13327-branch-2-001.patch

Patch 001 of an output stream specification; covering what is meant to happen, and what really
does.

There's no tests for this, really the tests should be looking to see what the FS does on flush/hflush
and hsync and verify that it matches whatever the declared behaviour is; we'll add contract
test options for inconsistent metadata, writeback in chunks and flush-when-closed as options.

And of course, we'll need to have a test which probes the dest FS for data being written without
relying on just the getFileStatus() length, because of course, HDFS under-reports there while
the file is open. It'll need a stream.hflush() followed by open(path).read() != -1 as the
check for a file not being empty

> Add OutputStream + Syncable to the Filesystem Specification
> -----------------------------------------------------------
>
>                 Key: HADOOP-13327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13327
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13327-branch-2-001.patch
>
>
> Write down what a Filesystem output stream should do. While core the API is defined in
Java, that doesn't say what's expected about visibility, durability, etc —and Hadoop Syncable
interface is entirely ours to define.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message