airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject The need for LocalTaskJob
Date Sat, 04 Aug 2018 13:10:19 GMT
Hi Max, Dan et al,

Currently, when a scheduled task runs this happens in three steps: 

1. Worker 
2. LocalTaskJob
3. Raw task instance

It uses (by default) 5 (!) different processes:

1. Worker 
2. Bash + Airflow
3. Bash + Airflow 

I think we can merge worker and LocalTaskJob as the latter seems exist only to track a particular
task. This can be done within the worker without side effects. Next to thatI think we can
limit the amount of (airflow) processes to 2 if we remove the bash dependency. I don’t see
any reason to depend on bash.

Can you guys shed some light on what the thoughts were around those choices? Am I missing
anything on why they should exist?

Cheers
Bolke

Verstuurd vanaf mijn iPad
Mime
View raw message