Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Michael Crawford <michael.crawford@modernizingmedicine.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_7481B6EF-E00C-4BC4-87FB-3D6E833C09F6"
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: hiding aws secret key in connections
Date: Wed, 20 Sep 2017 18:47:19 -0400
References: <8B0A20F0-6554-4D9C-8842-A3D4DB774FE8@modernizingmedicine.com>
 <FB7617F7-6B2F-4072-AA3C-62A68CD77611@modernizingmedicine.com>
 <CA+0Vu45WwpPcydi5GaHiDkjUKs_1acOaFfOydVYSp9-hyOESvQ@mail.gmail.com>
To: dev@airflow.incubator.apache.org
In-Reply-To: <CA+0Vu45WwpPcydi5GaHiDkjUKs_1acOaFfOydVYSp9-hyOESvQ@mail.gmail.com>
Message-Id: <3CC4E15F-6359-4154-8A8A-D86F7B48772C@modernizingmedicine.com>
archived-at: Wed, 20 Sep 2017 22:47:33 -0000

--Apple-Mail=_7481B6EF-E00C-4BC4-87FB-3D6E833C09F6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

After some more research it appears that aws_hook in contrib/hooks =
actually does do it the way I proposed with looking at the login and =
password of the connection, but it doesn=E2=80=99t look at the extra =
json for the access and secret keys.

def get_client_type(self, client_type, region_name=3DNone):
    try:
        connection_object =3D self.get_connection(self.aws_conn_id)
        aws_access_key_id =3D connection_object.login
        aws_secret_access_key =3D connection_object.password

        if region_name is None:
            region_name =3D =
connection_object.extra_dejson.get('region_name')

    except AirflowException:
        # No connection found: fallback on boto3 credential strategy
        # http://boto3.readthedocs.io/en/latest/guide/configuration.html
        aws_access_key_id =3D None
        aws_secret_access_key =3D None

    return boto3.client(
        client_type,
        region_name=3Dregion_name,
        aws_access_key_id=3Daws_access_key_id,
        aws_secret_access_key=3Daws_secret_access_key
    )
However the S3Hook looks for this info in a different using the older =
boto library instead of boto3. =20

So it appears we have 2 different parts of airflow interacting with aws =
but specifying their credentials in different ways.

Thoughts?


> On Sep 19, 2017, at 12:01 PM, Ali Uz <aliuz1@gmail.com> wrote:
>=20
> We use a dynamic config where we iterate through a JSON file, and all
> sensitive info (like api keys, aws keys, etc...) are pulled from a =
remote
> k/v store when airflow starts and adds them as fields to the JSON =
config
> file.
>=20
> On Tue, Sep 19, 2017 at 6:54 PM, Michael Crawford <
> michael.crawford@modernizingmedicine.com> wrote:
>=20
>> Did my message go through?  I have never tried to send an email to =
the
>> list before, only silently monitored.
>>=20
>> Does anyone have any ideas?   I would be happy to create an issue and =
code
>> up the fix myself, but I just wanted to ping here first to make sure =
I
>> wasn=E2=80=99t missing anything and try to get a consensus on how to =
handle this.
>>=20
>> Thanks,
>> Mike
>>=20
>>> On Sep 18, 2017, at 8:03 PM, Michael Crawford <michael.crawford@
>> modernizingmedicine.com> wrote:
>>>=20
>>> Hi,
>>>=20
>>> I was wondering if anything had ever been proposed for having the =
aws
>> secret key hidden in the aws type connection.
>>>=20
>>> Currently passing in these credentials is done by defining the some =
json
>> in the extra params section of the connection like
>>> {"aws_access_key_id":"_your_aws_access_key_id_",
>> "aws_secret_access_key": "_your_aws_secret_access_key_=E2=80=9D}
>>>=20
>>> While this does work it leaves the secret access key in plain text =
for
>> anyone that has access to the connections.
>>>=20
>>> I know there are other options about setting them as environment
>> variables, but this doesn=E2=80=99t help if we need to define more =
than one aws
>> connection with different access keys.
>>>=20
>>> Two things that immediately came to mind for how to do this:
>>>=20
>>> 1.  use login and password sections of the connection for the access =
and
>> secret keys so that the secret gets hidden and encrypted like all the =
other
>> passwords.
>>> 2. have an option to encrypt the extra params
>>>=20
>>> Option 1 seems most logical and should be too hard to implement.
>>>=20
>>> Open to any ideas people might have on this.
>>>=20
>>> Thanks,
>>> Mike
>>=20
>>=20


--Apple-Mail=_7481B6EF-E00C-4BC4-87FB-3D6E833C09F6--