beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <>
Subject [jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem
Date Fri, 15 Sep 2017 09:33:00 GMT


Steve Loughran commented on BEAM-2500:

bq. . So we'll have to have a way to stream bytes into S3 (some implementation of WrittableByteChannel).
I'm not sure if S3 client library already supports this.

yes, it takes an input stream through its xfer manager, but needs one supporting mark/restore
if you want the manager to handle a transient failure of the write of a block of data.

> Add support for S3 as a Apache Beam FileSystem
> ----------------------------------------------
>                 Key: BEAM-2500
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-extensions
>            Reporter: Luke Cwik
>            Priority: Minor
>         Attachments: hadoop_fs_patch.patch
> Note that this is for providing direct integration with S3 as an Apache Beam FileSystem.
> There is already support for using the Hadoop S3 connector by depending on the Hadoop
File System module[1], configuring HadoopFileSystemOptions[2] with a S3 configuration[3].
> 1:
> 2:
> 3:

This message was sent by Atlassian JIRA

View raw message