airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Timmins <ja...@astronomer.io>
Subject Re: extension of the REST API
Date Fri, 16 Oct 2020 04:17:13 GMT
Hi Franco,

I know it may seem strange that the API in 2.0 won't support uploading or
substantially modifying DAGs, (as you mentioned, there is an Update
endpoint, but it is limited to pausing/unpausing DAGs). This is because the
goal for the API, at least for now, is to have feature parity with the
Airflow UI and CLI. Since DAG uploading isn't supported by those tools,
it's out of scope for the 2.0 API. If you're curious about the goals and
decisions behind the API, there's more info in the improvement proposal.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-32%3A+Airflow+REST+API

While I haven't used any of the plugins, you could fork Airflow and add
that functionality to the API yourself if you'd like.

A reasonable next question is whether or not that functionality is planned
for a future release. I haven't heard anything about that on the project
roadmap, but others may have more insight there. In general, the focus has
been entirely on getting 2.0 stable and shipped. There are some features
planned for 2.1, but I don't think they're API related. Beyond that, we'll
have to see what features are needed by the community.

Hopefully that provides a bit of clarity into the confusing aspects of the
API.

Kind regards,
James


On Thu, Oct 15, 2020 at 2:33 PM Franco Peschiera <franco.peschiera@gmail.com>
wrote:

> wow, that's great! Thanks for the quick and positive response.
>
> One thing though. I did not find a way to write (POST) a DAG (i.e., upload
> a new DAG). Maybe for security reasons? (Although I see an "Update a DAG"
> endpoint).
>
> Thanks again.
>
> On Thu, Oct 15, 2020 at 11:19 PM Kaxil Naik <kaxilnaik@gmail.com> wrote:
>
>> Airflow 2.0 will have a full-featured API:
>> https://github.com/apache/airflow/blob/master/UPDATING.md#migration-guide-from-experimental-api-to-stable-api-v1
>>
>> API Spec & Details:
>> https://airflow.readthedocs.io/en/latest/stable-rest-api-ref.html
>>
>>
>> On Thu, Oct 15, 2020 at 10:09 PM Franco Peschiera <
>> franco.peschiera@gmail.com> wrote:
>>
>>> Hello everyone,
>>>
>>> We're currently building a web app that makes use of airflow to delegate
>>> tasks. First of all, thanks for this excellent tool: it seems it will save
>>> us a lot of time and headaches.
>>>
>>> I've been checking the REST api since we want to ideally communicate
>>> exclusively this way. And yes, I know that functionality appears to be new
>>> / recent (because of the "experimental" tag in the docs and the URL).
>>> Having said that, there are some things that the REST API doesn't do (yet?)
>>> that we would love to have: (1) upload a new DAG, (2) check the status of a
>>> dagrun, among others.
>>>
>>> The rest api docs I'm reading:
>>> https://airflow.apache.org/docs/stable/rest-api-ref.html
>>>
>>> On the other hand, I've found there are side projects / third party
>>> plugins that do offer this functionality:
>>> https://github.com/teamclairvoyant/airflow-rest-api-plugin
>>>
>>> So I have the following questions: (1) are there any plans on completing
>>> the official REST API to meet the CLI / python ones? (2) is it a good idea
>>> to try third party plugins for this? if so, do you recommend a specific one?
>>>
>>> Thanks again!
>>>
>>> Franco
>>>
>>>>

Mime
View raw message