airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] dazza-codes commented on issue #5788: [POC] multi-threading using asyncio
Date Thu, 02 Jan 2020 17:39:02 GMT
dazza-codes commented on issue #5788: [POC] multi-threading using asyncio
URL: https://github.com/apache/airflow/pull/5788#issuecomment-570281992
 
 
   > @ash recently made some changes to the LocalExecutor to use os.fork instead of processes,
this makes the multithreading in the LocalExecutor much faster and lighter weight.
   
   That's curious.  In python, the GIL effectively makes all threading run with preemptive
scheduling as though it is running in a single thread and the performance gains are almost
none and sometimes it can hurt performance because the system threads start thrashing to get
a hold of the GIL.  The threads do not run in parallel because of the GIL, so parallel performance
requires multiprocessing with a process pool.  I recently delved into a good article on this
topic, with a lot of good references for more details:
   - https://realpython.com/async-io-python
   - https://realpython.com/python-gil/
   - plus links to talks by David Beazley and the gilectomy etc.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message