flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5788) Document assumptions about File Systems and persistence
Date Tue, 14 Feb 2017 11:19:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865609#comment-15865609

ASF GitHub Bot commented on FLINK-5788:

Github user alpinegizmo commented on a diff in the pull request:

    --- Diff: docs/internals/filesystems.md ---
    @@ -0,0 +1,138 @@
    +title: "File Systems"
    +nav-parent_id: internals
    +nav-pos: 10
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +  http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing,
    +software distributed under the License is distributed on an
    +KIND, either express or implied.  See the License for the
    +specific language governing permissions and limitations
    +under the License.
    +* Replaced by the TOC
    +Flink has its own file system abstraction via the `org.apache.flink.core.fs.FileSystem`
    +This abstraction provides a common set of operations and minimal guarantees across various
    +of file system implementations.
    +The `FileSystem`'s set of available operations is quite limited, in order to suport a
    +range of file systems. For example, appending to or mutating existing files is not supported.
    +File systems are identified by a *file system scheme*, such as `file://`, `hdfs://`,
    +# Implementations
    +Flink implements the file systems directly, with the following file system schemes:
    +  - `file`, which represents the machines local file system.
    +Other file system types are accessed by an implementation that bridges to the suite of
file systems supported by
    +[Apache Hadoop](https://hadoop.apache.org/). The following is an incomplete list of examples:
    +  - `hdfs`: Hadoop Distributed File System
    +  - `s3`, `s3n`, and `s3a`: Amazon S3 file system
    +  - `gcs`: Google Cloud Storage
    +  - `maprfs`: The MapR distributed file system
    +  - ...
    +Flink loads Hadoop's file systems transparently if it finds the Hadoop File System classes
in the class path and finds a valid
    +Hadoop configuration. By default, it looks for the Hadoop configuration in the class
path. Alternatively, one can specify a
    +custom location via the configuration entry `fs.hdfs.hadoopconf`.
    +# Persistence Guarantees
    +These `FileSystem` and its `FsDataOutputStream` instances are used to persistently store
data, both for results of applications
    +and for fault tolerance and recovery. It is therefore crucial that the persistence semantics
of these streams are well defined.
    +## Definition of Persistence Guarantees
    +Data written to an output stream is considered persistent, if two requirements are met:
    +  1. **Visibility Requirement:** It must be guaranteed that all other processes, machines,
    +     virtual machines, containers, etc. that are able to access the file see the data
    +     when given the absolute file path. This requirement is similar to the *close-to-open*
    +     semantics defined by POSIX, but restricted to the file itself (by its absolute path).
    +  2. **Durability Requirement:** The file system's specific durability/persistence requirements
    +     must be met. These are specific to the particular file system. For example the
    +     {@link LocalFileSystem} does not provide any durability guarantees for crashes of
    +     hardware and operating system, while replicated distributed file systems (like HDFS)
    +     guarantee typically durability in the presence of up to concurrent failure or *n*
    +     nodes, where *n* is the replication factor.
    +Updates to the file's parent directory (such as that the file shows up when
    +listing the directory contents) are not required to be complete for the data in the file
    +to be considered persistent. This relaxation is important for file systems where updates
    +directory contents are only eventually consistent.
    +The `FSDataOutputStream` has to guarantee data persistence for the written bytes once
the call to
    +`FSDataOutputStream.close()` returns.
    +## Examples
    +  - For **fault-tolerant distributed file systems**, data is considered persistent once

    +    it has been received and acknowledged by the file system, typically by having been
    +    to a quorum of machines (*durability requirement*). In addition the absolute file
    +    must be visible to all other machines that will potentially access the file (*visibility
    +    Whether data has hit non-volatile storage on the storage nodes depends on the specific
    +    guarantees of the particular file system.
    +    The metadata updates to the file's parent directory are not required to have reached
    +    a consistent state. It is permissible that some machines see the file when listing
the parent
    +    directory's contents while other do not, as long as access to the file by its absolute
    --- End diff --
    while others do not

> Document assumptions about File Systems and persistence
> -------------------------------------------------------
>                 Key: FLINK-5788
>                 URL: https://issues.apache.org/jira/browse/FLINK-5788
>             Project: Flink
>          Issue Type: Improvement
>          Components: Documentation
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 1.3.0
> We should add some description about the assumptions we make for the behavior of {{FileSystem}}
implementations to support proper checkpointing and recovery operations.
> This is especially critical for file systems like {{S3}} with a somewhat tricky contract.

This message was sent by Atlassian JIRA

View raw message