airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gauri Kholkar <gpkholka...@gmail.com>
Subject Integrate Airflow with MySQL
Date Fri, 22 Sep 2017 07:58:26 GMT
I need to

1. run a select query on MYSQL DB and fetch the records.2. Records are
processed by python script.

Is xcom the way to go here? Also, MYSQLOperator only executes the query,
doesn't fetch the records. Is there any inbuilt transfer operator I can
use? How can I use a MYSQL hook here?

you may want to use a PythonOperator that uses the hook to get the data,
apply transformation and ship the (now scored) rows back some other place.

Can someone explain how to proceed regarding the same.

*Refer - http://markmail.org/message/x6nfeo6zhjfeakfe
<http://markmail.org/message/x6nfeo6zhjfeakfe>*

def do_work():
    mysqlserver = MySqlHook(connection_id)
    sql = "SELECT * from table where col > 100 "
    row_count = mysqlserver.get_records(sql, schema='testdb')
    print row_count[0][0]

callMYSQLHook = PythonOperator(
    task_id='fetch_from_testdb',
    python_callable=mysqlHook,
    dag=dag)

Is this the correct way to proceed?


Regards

Gauri

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message