airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Toonstra <gtoons...@gmail.com>
Subject Re: How to query a database and then send an email with the query results?
Date Sat, 11 Mar 2017 06:51:12 GMT
You can also generalize the tasks to make them more reusable:

1. an operator that runs a query and stores the result in a file in a
generally available location (for all workers).
2. an operator extending the email operator that pulls in the file from the
general location to the local worker and sends "a file".

this allows you to reuse the work in completely different contexts and also
send the file over email, FTP or simply leave them there.

You'd then pass the location of the file around instead of the contents.

Rgds,

Gerard


On Sat, Mar 11, 2017 at 12:19 AM, Jayesh Senjaliya <jhsonline@gmail.com>
wrote:

> Hi Anne,
>
> I would go with first approach unless the result can be very big, since all
> xcom results gets stored in airflow db.
>
> another approach would be to use MySQLOperator + EmailOperator within
> pythonOperator
> as a wrapper. you ll have to execute the operator manually in the code
> though.
>
> if nothing works, one can always fall back to bare python code.
>
> - Jayesh
>
>
>
> On Fri, Mar 10, 2017 at 2:34 PM, Anne Yeung <ayeung@squareup.com> wrote:
>
> > Hi Airflow community,
> >
> > Does anyone have a recommendation on how I run a database query and then
> > email the results in a nicely formatted HTML table to a list of emails?
> >
> > I'm thinking of using a MySQLOperator + EmailOperator and passing the
> > results between the two tasks using xcom, but I'm also thinking that I
> > could just use a PythonOperator and manually add in the db connection and
> > query. Anyone have recommendation on how they do it?
> >
> > Thanks,
> > Anne
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message