airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Edmiston <>
Subject Re: Airflow & AWS Lambda Tutorials?
Date Thu, 23 Aug 2018 21:13:30 GMT
Hey Andrew -

My suggestions are similar to Marcel's.

I favor invoking the Lambda function with the invoke
call in boto3 in a PythonOperator, or using the AwsHook if that abstraction
adds value here vs calling boto directly.

I second that using the SimpleHttpOperator in Airflow to hit the Lambda
function like a REST API is a reasonable alternative.

Personally, I would also recommend using a serverless framework such as
Zappa or Serverless as in my experience, there's a surprising amount of
boilerplate involved in writing, deploying, updating, and maintaining
serverless functions (+ API Gateway etc).

If you're considering Lambda alternatives, I think the Zeit Now Serverless
Docker Beta <> is a pretty cool and
very easy way to run serverless functions.  There's a Hello World Flask
example <>
that shows how it all works in just a few lines.

*Taylor Edmiston*
Blog <> | CV
<> | LinkedIn
<> | AngelList
<> | Stack Overflow

On Wed, Jul 11, 2018 at 3:07 PM Hamade, Marcel (GfK) <>

> The aws_lambda_hook it self uses the AWS_hook which is a wrapper arount
> the boto3 library(the standard way to interact with AWS via python). If you
> add the AWS connections correctly you can use the hook in one of your self
> written operators to trigger your lambda functions
> Ano way would be to trigger Lambda functions via AWS API Gateway
> For that you can use the HTTP-Operator as a starting point but of course
> you have to handle HTTP-Authorisation.
> Cheers,
> Marcel
> ________________________________________
> Von: Andrew Maguire <>
> Gesendet: Mittwoch, 11. Juli 2018 13:33
> An:
> Betreff: Airflow & AWS Lambda Tutorials?
> Hi all,
> Does anyone know of any tutorials or examples of triggering an aws lambda
> function from airflow?
> I googled a bit bit could not find much. I do see the aws_lambda_hook in
> there but just a little unsure how to use it.
> Cheers,
> Andy
> ________________________________
> GfK SE, Nuremberg, Germany, commercial register at the local court
> Amtsgericht Nuremberg HRB 25014; Management Board: Peter Feld (CEO),
> Christian BigatĂ  Joseph (CFO). Supervisory Board Chairman: Ralf
> Klein-Bölting. This email and any attachments may contain confidential or
> privileged information. Please note that unauthorized copying, disclosure
> or distribution of the material in this email is not permitted.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message