airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-1520) S3Hook uses boto2
Date Mon, 05 Feb 2018 13:16:00 GMT


ASF subversion and git services commented on AIRFLOW-1520:

Commit 82a65eeca9bc654cec4d0f356c3db70bc8ab6838 in incubator-airflow's branch refs/heads/master
from [~baynham]
[;h=82a65ee ]

[AIRFLOW-1968][AIRFLOW-1520] Add role_arn and aws_account_id/aws_iam_role support back to
aws hook

In PR2532 (AIRFLOW-1520), the AWS credential code
was refactored into a general
 AWS hook.  When that change was made, the existing
assume role code was
 removed, leaving only ID/Secret credentials as an
option.  Our dags rely on
 role assumption to access external S3 buckets, so
this code re-adds role
 assumption via STS.

Additionally, in order to make this a bit easier,
I changed _get_credentials to
 return a functioning boto3 session which is used
by the public methods to
 initialize clients/resources/whatever.  This
seemed a better route than
 adding another returnval in an already long list.

Closes #2918 from CannibalVox/aws_hook_support_sts

> S3Hook uses boto2
> -----------------
>                 Key: AIRFLOW-1520
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Niels Zeilemaker
>            Priority: Major
>             Fix For: 1.9.0
> The S3Hook uses boto2 which does not support container roles. Therefore, we need to add
permissions to the underlying ec2 instances instead.
> Upgrading to boto3 fixes this

This message was sent by Atlassian JIRA

View raw message