airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Big change in the way we run tests (AIP-10 completed)
Date Thu, 18 Jul 2019 13:21:58 GMT
Awesome!

Sent from my iPhone

> On 18 Jul 2019, at 14:38, Dan Davydov <ddavydov@twitter.com.invalid> wrote:
> 
> This is amazing! Testing locally has always been quite painful. Thank you!
> 
> On Wed, Jul 17, 2019 at 11:54 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
> 
>> Hello everyone,
>> 
>> I would like to announce that I just merged the last (3/3) part of AIP-10 -
>> big change in the way we run CI tests for Airflow.
>> 
>> Nothing changes in the tests, but the way we run them in Travis CI and
>> locally is quite different. You should be able to reproduce Travis CI tests
>> super-easily on your local Linux/MacOS development machine now following
>> the documentation
>> <
>> https://github.com/apache/airflow/blob/master/CONTRIBUTING.md#integration-test-development-environment
>>> 
>> .
>> The change gets rid of tox, Dockerises everything including
>> pylint/mypy/flake checks, and optimises it so that it can be conveniently
>> run both in CI and locally. Including faster iterations for re-running
>> tests several times.
>> 
>> Travis CI builds are now better and show python version + they print
>> summary of failed tests at the end of the log. We also have a way to run
>> static code check super-easily locally see .
>> 
>> I truly hope it will make it easier for casual contributors to contribute
>> to Airflow but also a lot of experienced Airflowers might find it really
>> helpful. Please rebase to master and make a good use of it.
>> 
>> The CONTRIBUTING.md documentation
>> <
>> https://github.com/apache/airflow/blob/master/CONTRIBUTING.md#integration-test-development-environment
>>> 
>> has changed a lot (and hopefully vastly improved).I expect some (hopefully
>> minor) teething problems - in case you have any troubles please follow the
>> troubleshooting
>> <
>> https://github.com/apache/airflow/blob/master/CONTRIBUTING.md#troubleshooting
>>> 
>> section of the documentation - if you cannot solve it following it, reach
>> out to me on Slack or here in the discussion. I will try to be available as
>> much as I can over the next few days to help in case of problems.
>> 
>> There are few follow-up PRs/Proposal that are in the making for which
>> AIP-10 was prerequisite (Breeze - simplified development environment,
>> Pre-commit hooks) - also we will try to migrate out of Travis CI because of
>> the recent problems with it (AIP-10 was instrumental to enable fast
>> migration out of Travis) .
>> 
>> IMPORTANT! Together with Ash we are going to run webinar/screencast next
>> week (*Thursday 25th of July, 6.30pm CEST, 5.30pm London, 12:30 pm NYC,
>> 9.30am California*). It will present what's already merged in AIP -10 and
>> some of the improvements coming (Breeze/pre-commit). There will be separate
>> announcement tomorrow about it so stay tuned.
>> 
>> Big thanks to everyone who helped with this (especially to Ash for thorough
>> reviews and great comments, but also to Fokko, Kamil, Gerardo for their
>> support and comments - especially those critical ones that helped to make
>> it better. Also Daniel, Stefan for testing during last two weeks. It's
>> great community we have here :).
>> 
>> BTW. I think this was the longest PR I ever worked on. I wrote the first
>> version of the proposal
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+CI+image
>>> 
>> mid-January
>> and I worked on it since then.
>> Fun fact -  I hold now the 2nd place for PR with the biggest number of
>> comments (278)  in Airflow right after "[AIRFLOW-3585] - Add edges to
>> database" (284).
>> 
>> J.
>> 
>> 
>> 
>> --
>> 
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> 
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>> 

Mime
View raw message