airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nadeem Ahmed Nazeer <naz...@neon-lab.com>
Subject Re: S3Hook and connection
Date Tue, 19 Jul 2016 17:04:22 GMT
Thanks for the response Paul. Crypto package would be my last resort. But
it would still not solve (2) where I am looking to create the connection
automatically.

Awaiting further answers..

Thanks,
Nadeem

On Tue, Jul 19, 2016 at 9:03 AM, Paul Minton <pminton@change.org> wrote:

> Nadeem, this doesn't directly answer either 1) or 2), but have you
> considered using the option "is_exrra_encrypted"? This encrypts the extra
> json as it would for the rest of the credentials on the connection object
> (ie using a fernet key and the encryption package)
>
> On Mon, Jul 18, 2016 at 10:00 PM, Nadeem Ahmed Nazeer <nazeer@neon-lab.com
> >
> wrote:
>
> > Hi,
> >
> > Appreciate if someone could please provide assistance on this.
> >
> > Thanks,
> > Nadeem
> >
> > On Fri, Jul 15, 2016 at 4:15 PM, Nadeem Ahmed Nazeer <
> nazeer@neon-lab.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We are using the S3Hook in several of our airflow TI's to read and
> write
> > > data from S3.
> > >
> > > We are creating a s3 connection from the UI where we choose the below
> > > options.
> > >
> > > Conn Id - s3
> > > Conn Type - S3
> > > Extra - {"aws_access_key_id":"key", "aws_secret_access_key": "key"}
> > >
> > > In pipeline code we use this connection as below,
> > >
> > > s3 = S3Hook(s3_conn_id='s3')
> > >
> > > We are looking into other options to define this connection as it is a
> > > security issue to have the keys open like this. We tried defining the
> > > connection id and connection type alone in the UI without the keys. In
> > this
> > > case, the tasks that read from S3 succeed but the ones that delete or
> > > create files/objects fail with '403 Forbidden' error from S3. Did some
> > > digging in the S3_Hook code and found that if the keys are not in the
> > Extra
> > > parameter then it would use the boto config but that doesn't seem to
> work
> > > in my case for reasons I am unable to find.
> > >
> > > All our other python scripts interact with S3 using the boto config on
> > the
> > > system without any problems.
> > >
> > > 1)
> > > Need help on why the s3 hook isn't using the boto config. Am I missing
> to
> > > pass some other parameters to this connection?
> > >
> > > 2)
> > > How to define the s3 connection as environmental variable? We are
> > > installing airflow via Chef and would want to have an environmental
> > > variable like AIRFLOW_CONN_S3 created for this connection so that we
> > don't
> > > have to manually do it in the UI every time we run the setup.
> > >
> > > Documentation says, it has the connection has to be in a URI format. On
> > > S3, I could access different buckets with the same connection. But
> since
> > it
> > > has to be in URI format, does that mean i create one connection per
> > bucket
> > > and use it? Did not find any examples of this anywhere hence asking.
> > >
> > > Thanks,
> > > Nadeem
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message