airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] kaxil edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability
Date Thu, 12 Sep 2019 11:52:44 GMT
kaxil edited a comment on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized DAG in
DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-530789821
 
 
   Few things to do:
   
   - [ ] Add https://github.com/astronomer/airflow/commit/baf12f626e6d56dfde735faaed71b2c30cb4befb
and add tests for it
   - [ ] Reduce the info we store in Serialized DAGs by removing all the default arguments
that are not overridden by users. Eg `owner` in DAG & Task etc. This will help reduce
blob size as well as reduce the time spent in `_deserialise` method. 
   - [ ] Agree / dis-agree on using https://pypi.org/project/SQLAlchemy-JSONField/ instead
of our code . It also has a nice option of specifying json library as compared to providing
that info in the `create_engine.json_serializer` and `create_engine.json_deserializer` parameters
in  https://docs.sqlalchemy.org/en/13/core/type_basics.html#sqlalchemy.types.JSON
   
   cc @coufon @ashb 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message