airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Modeling rate limited api calls in airflow
Date Thu, 09 Aug 2018 08:25:17 GMT

I am in the process of migrating a bespoke data pipe line built around celery into airflow.

We have a number of different tasks which interact with the Adwords API which has a rate limiting
policy. The policy isn't a fixed number of requests its variable.

In our celery code we have handled this by capturing a rate limit error response and setting
a key in redis to make sure that no tasks execute against the API until it's expired. Any
task that does get executed checks for the presence of the key and if the key exists issues
a retry for when the rate limit is due to expire.

Moving over to Airflow I can't find a way to go about scheduling a task to retry in a specific
amount of time. Doing some reading it seems a Sensor could work to prevent other dags from
executing whilst the rate limit is present.

I also can't seem to find an example of handling different exceptions from a python task and
adapting the retry logic accordingly.

Any pointers would be much appreciated,


View raw message