Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 13 Jul 2017 12:43:00 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13085598.1499461506000.238978.1499949780942@Atlassian.JIRA>
In-Reply-To: <JIRA.13085598.1499461506000@Atlassian.JIRA>
References: <JIRA.13085598.1499461506000@Atlassian.JIRA> <JIRA.13085598.1499461506698@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-2572) Implement an S3 filesystem for
 Python SDK
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 13 Jul 2017 12:43:05 -0000


    [ https://issues.apache.org/jira/browse/BEAM-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085648#comment-16085648 ] 

Steve Loughran commented on BEAM-2572:
--------------------------------------

bear in mind that credentials are highly sensitive facts, which mustn't leak into logs, stack traces, bug reports. If credentials are to be passed around this way, make sure that they are never visible.

note also that if you are running on EC2, you get session credentials for free from the IAM service; all you need to do is set up the auth chain right.

> Implement an S3 filesystem for Python SDK
> -----------------------------------------
>
>                 Key: BEAM-2572
>                 URL: https://issues.apache.org/jira/browse/BEAM-2572
>             Project: Beam
>          Issue Type: Task
>          Components: sdk-py
>            Reporter: Dmitry Demeshchuk
>            Assignee: Ahmet Altay
>            Priority: Minor
>
> There are two paths worth exploring, to my understanding:
> 1. Sticking to the HDFS-based approach (like it's done in Java).
> 2. Using boto/boto3 for accessing S3 through its common API endpoints.
> I personally prefer the second approach, for a few reasons:
> 1. In real life, HDFS and S3 have different consistency guarantees, therefore their behaviors may contradict each other in some edge cases (say, we write something to S3, but it's not immediately accessible for reading from another end).
> 2. There are other AWS-based sources and sinks we may want to create in the future: DynamoDB, Kinesis, SQS, etc.
> 3. boto3 already provides somewhat good logic for basic things like reattempting.
> Whatever path we choose, there's another problem related to this: we currently cannot pass any global settings (say, pipeline options, or just an arbitrary kwarg) to a filesystem. Because of that, we'd have to setup the runner nodes to have AWS keys set up in the environment, which is not trivial to achieve and doesn't look too clean either (I'd rather see one single place for configuring the runner options).
> Also, it's worth mentioning that I already have a janky S3 filesystem implementation that only supports DirectRunner at the moment (because of the previous paragraph). I'm perfectly fine finishing it myself, with some guidance from the maintainers.
> Where should I move on from here, and whose input should I be looking for?
> Thanks!


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)