airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Davis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2127) Airflow's Alembic migrations globally disable logging
Date Tue, 20 Feb 2018 19:20:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370492#comment-16370492
] 

Matt Davis commented on AIRFLOW-2127:
-------------------------------------

Here's the PR adding {{disable_existing_loggers=False}}: https://github.com/apache/incubator-airflow/pull/3059

> Airflow's Alembic migrations globally disable logging
> -----------------------------------------------------
>
>                 Key: AIRFLOW-2127
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2127
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db
>            Reporter: Matt Davis
>            Priority: Major
>
> When running Airflow's {{[upgradedb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L295]}},
{{[resetdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L311]}},
and {{[initdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L83]}} functions
logging is disabled thereafter for all but the {{sqlalchemy.engine}} and {{alembic}} loggers.
This is caused [this usage|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]
of Python's {{fileConfig}} function, which by default disables all loggers that aren't part
of the supplied configuration. (See [Python 2 docs|https://docs.python.org/2/library/logging.config.html#logging.config.fileConfig]
and [Python 3 docs|https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig].)
This can be fixed by adding {{disable_existing_loggers=False}} to the call of {{fileConfig}}.
> This has affected us at Clover Health because we use these database utility functions
in some of our tooling, and particularly our _tests_ of the tooling. Having all logging disabled
in the midst of our tests has it difficult to test our use of logging in completely unrelated
parts of our codebase.
> As an example, we were trying to use [pytest's caplog feature|https://docs.pytest.org/en/latest/logging.html#caplog-fixture],
but were unable to do so with logging globally disabled by {{fileConfig}}. Here's an example
of a test that fails with {{disable_existing_loggers=True}} (the default), but passes with
{{disable_existing_loggers=False}}.
> {code}
> import logging
> import pytest
> import airflow.utils.db as af_db
> LOGGER = logging.getLogger(__name__)
> @pytest.fixture(autouse=True)
> def resetdb():
>     af_db.resetdb()
> def test_caplog(caplog):
>     LOGGER.info('LINE 1')
>     assert caplog.record_tuples
>     assert 'LINE 1' in caplog.text
> {code}
> I'll submit a pull request shortly to add {{disable_existing_loggers=False}} to Airflow's
{{[env.py|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message