beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2868) Allow FileSystems.copy() to copy between different scheme
Date Tue, 12 Sep 2017 19:47:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163539#comment-16163539
] 

Luke Cwik commented on BEAM-2868:
---------------------------------

Currently FileSystems.copy() enforces that the same scheme is used because it just forwards
the call to a specific file system implementation.To support this feature request you want
to split the list of copy requests into ones which are for the same scheme and ones which
are for different schemes. For the ones in the different scheme you effectively want to have
some support code which uses FileSystems.create(...) and FileSystems.read(...) and perform
a ByteChannel copy. Typically you want to perform these copies in parallel to mitigate the
overhead of copying small files.

> Allow FileSystems.copy() to copy between different scheme
> ---------------------------------------------------------
>
>                 Key: BEAM-2868
>                 URL: https://issues.apache.org/jira/browse/BEAM-2868
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Reza ardeshir rokni
>            Assignee: Luke Cwik
>            Priority: Minor
>
> It would be useful to allow FileSystem.copy() to be able to copy from different scheme.
For example when copying files from a Object store to the worker. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message