airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] feluelle commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD
Date Tue, 15 Oct 2019 08:55:21 GMT
feluelle commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation
from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334823173
 
 

 ##########
 File path: BREEZE.rst
 ##########
 @@ -474,214 +617,213 @@ Run pylint checks for all files:
      ./breeze --static-check-all-files pylint
 
 
-The ``license`` check is also run via separate script and separate docker image containing
+The ``license`` check is run via a separate script and a separate docker image containing
the 
 Apache RAT verification tool that checks for Apache-compatibility of licences within the
codebase.
-It does not take pre-commit parameters as extra args.
+It does not take pre-commit parameters as extra arguments.
 
 .. code-block:: bash
 
      ./breeze --static-check-all-files licenses
 
-Building the documentation
---------------------------
+Running Static Code Checks from the Host
+----------------------------------------
 
-The documentation is build using ``-O``, ``--build-docs`` command:
+You can trigger the static checks from the host environment, without entering the Docker
container. To do this, run the following scripts (the same is done in Travis CI):
 
-.. code-block:: bash
+* `<scripts/ci/ci_check_license.sh>`_ - checks the licenses.
+* `<scripts/ci/ci_docs.sh>`_ - checks that documentation can be built without warnings.
+* `<scripts/ci/ci_flake8.sh>`_ - runs Flake8 source code style enforcement tool.
+* `<scripts/ci/ci_lint_dockerfile.sh>`_ - runs lint checker for the Dockerfile.
+* `<scripts/ci/ci_mypy.sh>`_ - runs a check for mypy type annotation consistency.
+* `<scripts/ci/ci_pylint_main.sh>`_ - runs pylint static code checker for main files.
+* '`<scripts/ci/ci_pylint_tests.sh>`_ - runs pylint static code checker for tests.
 
-     ./breeze --build-docs
+The scripts may ask you to rebuild the images, if needed.
 
-Results of the build can be found in ``docs/_build`` folder. Often errors during documentation
generation
-come from the docstrings of auto-api generated classes. During the docs building auto-api
generated
-files are stored in ``docs/_api`` folder - so that in case of problems with documentation
you can
-find where the problems with documentation originated from.
+You can force rebuilding the images by deleting the [.build](./build) directory. This directory
keeps cached
+information about the images already built and you can safely delete it if you want to start
from scratch.
 
-Running tests directly from host
---------------------------------
+After documentation is built, the html results are available in the [docs/_build/html](docs/_build/html)
folder.
+This folder is mounted from the host so you can access those files on your host as well.
 
-If you wish to run tests only and not drop into shell, you can run them by providing
--t, --test-target flag. You can add extra nosetest flags after -- in the commandline.
+Running Static Code Checks in the Docker
+------------------------------------------
 
-.. code-block:: bash
+If you are already in the Breeze Docker environment (by running the ``./breeze`` command),
you can also run the same static checks from the container:
 
-     ./breeze --test-target tests/hooks/test_druid_hook.py -- --logging-level=DEBUG
+* Mypy: ``./scripts/ci/in_container/run_mypy.sh airflow tests``
+* Pylint for main files: ``./scripts/ci/in_container/run_pylint_main.sh``
+* Pylint for test files: ``./scripts/ci/in_container/run_pylint_tests.sh``
+* Flake8: ``./scripts/ci/in_container/run_flake8.sh``
+* Licence check: ``./scripts/ci/in_container/run_check_licence.sh``
+* Documentation: ``./scripts/ci/in_container/run_docs_build.sh``
 
-You can run the whole test suite with special '.' test target:
+Running Static Code Analysis for Selected Files
+-----------------------------------------------
 
-.. code-block:: bash
+In all static check scripts, both in the container and host versions, you can also pass a
module/file path as
+parameters of the scripts to only check selected modules or files. For example:
 
-    ./breeze --test-target .
+In the Docker container:
 
-You can also specify individual tests or group of tests:
+.. code-block::
 
-.. code-block:: bash
+  ./scripts/ci/in_container/run_pylint.sh ./airflow/example_dags/
 
-    ./breeze --test-target tests.core:TestCore
+or
 
-Pulling the latest images
--------------------------
+.. code-block::
 
-Sometimes the image on DockerHub is rebuilt from the scratch. This happens for example when
there is a
-security update of the python version that all the images are based on.
-In this case it is usually faster to pull latest images rather than rebuild them
-from the scratch.
+  ./scripts/ci/in_container/run_pylint.sh ./airflow/example_dags/test_utils.py
 
-You can do it via ``--force-pull-images`` flag to force pull latest images from DockerHub.
+On the host:
 
-In the future Breeze will warn you when you are advised to force pull images.
+.. code-block::
 
-Running commands inside Docker
-------------------------------
+  ./scripts/ci/ci_pylint.sh ./airflow/example_dags/
 
-If you wish to run other commands/executables inside of Docker environment you can do it
via
-``-x``, ``--execute-command`` flag. Note that if you want to add arguments you should specify
them
-together with the command surrounded with " or ' or pass them after -- as extra arguments.
 
-.. code-block:: bash
+.. code-block::
 
-     ./breeze --execute-command "ls -la"
+  ./scripts/ci/ci_pylint.sh ./airflow/example_dags/test_utils.py
 
-.. code-block:: bash
+Running Test Suites via Scripts
+--------------------------------------------
 
-     ./breeze --execute-command ls -- --la
+To run all tests with default settings (python 3.6, sqlite backend, docker environment),
enter:
 
+.. code-block::
 
-Running Docker Compose commands
--------------------------------
+  ./scripts/ci/local_ci_run_airflow_testing.sh
 
-If you wish to run docker-compose command (such as help/pull etc. ) you can do it via
-``-d``, ``--docker-compose`` flag. Note that if you want to add extra arguments you should
specify them
-after -- as extra arguments.
 
-.. code-block:: bash
+To select a Python version, backend, and a docker environment, specify:
 
-     ./breeze --docker-compose pull -- --ignore-pull-failures
+.. code-block::
 
-Using your host IDE
-===================
+  PYTHON_VERSION=3.5 BACKEND=postgres ENV=docker ./scripts/ci/local_ci_run_airflow_testing.sh
+
+To run kubernetes tests, enter:
+
+.. code-block::
+
+  KUBERNETES_VERSION==v1.13.0 KUBERNETES_MODE=persistent_mode BACKEND=postgres ENV=kubernetes
\
+    ./scripts/ci/local_ci_run_airflow_testing.sh
+
+* PYTHON_VERSION is one of 3.5/3.6/3.7
+* BACKEND is one of postgres/sqlite/mysql
+* ENV is one of docker/kubernetes/bare
+* KUBERNETES_VERSION is required for Kubernetes tests. Currently, it is KUBERNETES_VERSION=v1.13.0.
+* KUBERNETES_MODE is a mode of kubernetes: either persistent_mode or git_mode.
+
+Using Your Host IDE with Breeze
+===============================
 
 Configuring local virtualenv
 ----------------------------
 
-In order to use your host IDE (for example IntelliJ's PyCharm/Idea) you need to have virtual
environments
-setup. Ideally you should have virtualenvs for all python versions that Airflow supports
(3.5, 3.6, 3.7).
-You can create the virtualenv using ``virtualenvwrapper`` - that will allow you to easily
switch between
-virtualenvs using workon command and mange your virtual environments more easily.
+To use your host IDE (for example, IntelliJ's PyCharm/Idea), you need to set up virtual environments.
Ideally, you should have virtualenvs for all Python versions supported by Airflow (3.5, 3.6,
3.7).
+You can create a virtualenv using ``virtualenvwrapper``. This allows you to easily switch
between
+virtualenvs using the ``workon`` command and manage your virtual environments more easily.
 
 Typically creating the environment can be done by:
 
 .. code-block:: bash
 
   mkvirtualenv <ENV_NAME> --python=python<VERSION>
 
-After the virtualenv is created, you must initialize it. Simply enter the environment
-(using workon) and once you are in it run:
+After the virtualenv is created, you need to initialize it. Simply enter the environment
by 
+using ``workon`` and, once you are in it, run:
 
 .. code-block:: bash
 
   ./breeze --initialize-local-virtualenv
 
-Once initialization is done, you should select the virtualenv you initialized as the project's
default
+Once initialization is done, select the virtualenv you initialized as a default project
 virtualenv in your IDE.
 
-Running unit tests via IDE
+Running Unit Tests via IDE
 --------------------------
 
-After setting it up - you can use the usual "Run Test" option of the IDE and have all the
+When setup is done, you can use the usual **Run Test** option of the IDE, have all the
 autocomplete and documentation support from IDE as well as you can debug and click-through
-the sources of Airflow - which is very helpful during development. Usually you also can run
most
-of the unit tests (those that do not require prerequisites) directly from the IDE:
+the sources of Airflow, which is very helpful during development. Usually you can also run
most
+of the unit tests (those that do not have dependencies) directly from the IDE:
 
 Running unit tests from IDE is as simple as:
 
 .. image:: images/running_unittests.png
     :align: center
     :alt: Running unit tests
 
-Some of the core tests use dags defined in ``tests/dags`` folder - those tests should have
+Some of the core tests use dags defined in ``tests/dags`` folder. Those tests should have
 ``AIRFLOW__CORE__UNIT_TEST_MODE`` set to True. You can set it up in your test configuration:
 
 .. image:: images/airflow_unit_test_mode.png
     :align: center
     :alt: Airflow Unit test mode
 
 
-You cannot run all the tests this way - only unit tests that do not require external dependencies
-such as postgres/mysql/hadoop etc. You should use
-`Running tests in Airflow Breeze <#running-tests-in-airflow-breeze>`_ in order to run
those tests. You can
-still use your IDE to debug those tests as explained in the next chapter.
+You cannot run all the tests this way but only unit tests that do not require external dependencies
+such as Postgres/Mysql/Hadoop/etc. You should use the
 
 Review comment:
   ```suggestion
   such as Postgres/MySQL/Hadoop/etc. You should use the
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message