airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hamade, Marcel (GfK)" <Marcel.Ham...@gfk.com>
Subject Development flow for custom hook & operator (MongoDB)
Date Mon, 07 Aug 2017 08:18:13 GMT
Dear all,
I am not super experienced with python, please bear with me! Since I was not able to find
an existing MongoDB-Hook and Operator for airflow I decided to take things into my own hand.
I found out that I need to define a Hook which inherits from the BaseHook and implements some
functions. Then I would create an operator that uses this hook within a DAG and provide the
db, collection and query to it in order to get the results back in my DAG.

When I put my custom hook into the "/home/airflow/airflow/plugins" folder the operator is
not able to recognize the MongoDbHook-Class, also not when I put it into the “/usr/lib/python2.7/site-packages/airflow/hooks”
folder and restart all three airflow components.

I found this unanswered question, which is directly related to my question: https://groups.google.com/forum/#!msg/airbnb_airflow/O_FLCzWaZf8/qYuj0GzpDAAJ

My questions:
(1) How does the development flow look like when developing a custom hook?
(2) Where do I put what in order for airflow to find it?
(3) Is the MongoDB-Hook planned for the future? If so, can you estimate when?


________________________________

GfK SE, Nuremberg, Germany, commercial register at the local court Amtsgericht Nuremberg HRB
25014; Management Board: Peter Feld (CEO), Christian Bigatà Joseph (CFO), David Krajicek,
Alessandra Cama. Supervisory Board Chairman: Ralf Klein-Bölting. This email and any attachments
may contain confidential or privileged information. Please note that unauthorized copying,
disclosure or distribution of the material in this email is not permitted.
Mime
View raw message