airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow on ECS
Date Thu, 02 Nov 2017 18:06:03 GMT
Please remember that with the LocalExecutor your tasks run in process(group) with the scheduler.
If you want to restart the scheduler, it will need to wait until all tasks have finished that
are currently running. In addition if you tasks are resource intensive (cpu, memory) this
can also affect the scheduler. In 1.9.0 we are a little bit more robust in this respect, but
guarding against OOM errors is very hard.

Furthermore, the new logging framework in 1.9.0, will allow you to have logs centrally which
might be convenient. However, documentation is not up to date so you will have to tune it
yourself. 

My 2 cents,

Bolke.

> On 2 Nov 2017, at 18:55, Shoumitra Srivastava <shoumitra362@gmail.com> wrote:
> 
> Hi guys,
> 
> So far we have had a lot of success testing out Airflow and we are now
> going for a full scale deployment. To that end, we are considering
> dockerizing airflow and deploying it on one of our ECS clusters. We are
> planning on separating out the web server and the scheduler to separate
> tasks and using local executor with an RDS postgres and redis backend. Does
> anyone else have any suggestions regarding the setup? Any design patterns
> or good practises and gotchas would be welcome.
> 
> -Shoumitra


Mime
View raw message