airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arthur Wiedmer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-115) Migrate and Refactor AWS integration to use boto3 and better structured hooks
Date Fri, 13 May 2016 21:13:12 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arthur Wiedmer updated AIRFLOW-115:
-----------------------------------
    Description: 
h2. Current State

The current AWS integration is mostly done through the S3Hook, which uses non standard credentials
parsing on top of using boto instead of boto3 which is the current supported AWS sdk for Python.

h2. Proposal

an AWSHook should be provided that maps Airflow connections to the boto3 API. Operators working
with s3, as well as other AWS services would then inherit from this hook but extend the functionality
with service specific methods like get_key for S3, start_cluster for EMR, enqueue for SQS,
send_email for SES etc...

* AWSHook
**S3Hook
**EMRHook
**SQSHook
**SESHook
...


 

  was:
h2. Current State

The current AWS integration is mostly done through the S3Hook, which uses non standard credentials
parsing on top of using boto instead of boto3 which is the current supported AWS sdk for Python.

h2. Proposal

an AWSHook should be provided that maps Airflow connections to the boto3 API. Operators working
with s3, as well as other AWS services would then inherit from this hook but extend the functionality
with service specific methods like get_key for S3, start_cluster for EMR, enqueue for SQS,
send_email for SES etc...

AWSHook
  |_S3Hook
  |_EMRHook
  |_SQSHook
  |_SESHook
...


 


> Migrate and Refactor AWS integration to use boto3 and better structured hooks
> -----------------------------------------------------------------------------
>
>                 Key: AIRFLOW-115
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-115
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: AWS, boto3, hooks
>            Reporter: Arthur Wiedmer
>            Priority: Minor
>
> h2. Current State
> The current AWS integration is mostly done through the S3Hook, which uses non standard
credentials parsing on top of using boto instead of boto3 which is the current supported AWS
sdk for Python.
> h2. Proposal
> an AWSHook should be provided that maps Airflow connections to the boto3 API. Operators
working with s3, as well as other AWS services would then inherit from this hook but extend
the functionality with service specific methods like get_key for S3, start_cluster for EMR,
enqueue for SQS, send_email for SES etc...
> * AWSHook
> **S3Hook
> **EMRHook
> **SQSHook
> **SESHook
> ...
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message