airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan K Chin" <akc...@us.ibm.com>
Subject Adding Apache Airflow support to Elyra's visual pipeline editor
Date Wed, 28 Oct 2020 20:48:43 GMT
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt"
><div dir="ltr" ><div dir="ltr" style="outline: none; color: rgb(18, 18, 18);
font-family: Arial, Helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align:
start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing:
0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style:
initial; text-decoration-color: initial;" >Hello All,&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Our team is building a Jupyter lab extension to enable users to construct a
ai/data pipeline workflow with&nbsp;notebooks and python scripts using a visual editor.
We have been exploring how to add Airflow(awesome) to our list of pipeline/workflow runtimes.(<a
href="https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html" rel="noopener" style="outline:
none; position: relative; color: rgb(65, 120, 190);" target="_blank" >https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html</a>).
In a nutshell, we construct a dag using the information provided by the user in the visual
editor with a help of a jinja template, however the current experimental REST API nor the
one in 2.0 will have the ability to upload one to a dag directory. Have a work in progress
branch here:&nbsp;<a href="https://github.com/elyra-ai/elyra/pull/490" rel="noopener"
style="outline: none; position: relative; color: rgb(65, 120, 190);" target="_blank" >https://github.com/elyra-ai/elyra/pull/490</a></div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Our current implementation pushes&nbsp;the constructed dag to an S3 bucket
and syncs the bucket to airflow's dag dir&nbsp;via cron.&nbsp;A more official and
formal&nbsp;ability to sync to dags to a remote repository is something that would be
very useful and has been explored in:&nbsp;<a href="https://github.com/apache/airflow/pull/3138"
rel="noopener" style="outline: none; position: relative; color: rgb(65, 120, 190);" target="_blank"
>https://github.com/apache/airflow/pull/3138</a>&nbsp;as well as&nbsp;<span
style="outline: none; box-sizing: border-box;" >AIP-5<span>&nbsp;</span></span>.
Are there any plans to revive this effort post 2.0 stability?&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Its also been suggested that current best practice(is it?) should be to take
advantage of git-syncing the dag bag to a git repo. We are exploring this option as well.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Any comments or&nbsp;input would be greatly appreciated!</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" >--<br style="outline: none;" >Alan
Chin<br style="outline: none;" >CODAIT, San Francisco<br style="outline: none;" >Email
- akchin@us.ibm.com</div></div></div></div><BR>


Mime
View raw message