airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2127) Airflow's Alembic migrations globally disable logging
Date Sun, 25 Feb 2018 11:23:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376044#comment-16376044
] 

ASF subversion and git services commented on AIRFLOW-2127:
----------------------------------------------------------

Commit 496d0f4372a2eead616b8074db357db2b71c28eb in incubator-airflow's branch refs/heads/master
from [~jiffyclub]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=496d0f4 ]

[AIRFLOW-2127] Keep loggers during DB migrations

Python's logging.config.fileConfig function will,
by default,
disable all existing loggers when it is called.
The fileConfig
function is used with default arguments by Airflow
during
Alembic migrations and disables all loggers except
those from
alembic and sqlalchemy (this includes disabling
Airflow's own loggers).
This change sets the disable_existing_loggers flag
of fileConfig to
False so that it _does not_ disable any existing
loggers, allowing them
to continue working as normal.

See more in https://issues.apache.org/jira/browse/
AIRFLOW-2127.

Closes #3059 from jiffyclub/AIRFLOW-2127
/fileconfig-keep-loggers


> Airflow's Alembic migrations globally disable logging
> -----------------------------------------------------
>
>                 Key: AIRFLOW-2127
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2127
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db
>            Reporter: Matt Davis
>            Priority: Major
>             Fix For: 1.10.0
>
>
> When running Airflow's {{[upgradedb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L295]}},
{{[resetdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L311]}},
and {{[initdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L83]}} functions
logging is disabled thereafter for all but the {{sqlalchemy.engine}} and {{alembic}} loggers.
This is caused [this usage|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]
of Python's {{fileConfig}} function, which by default disables all loggers that aren't part
of the supplied configuration. (See [Python 2 docs|https://docs.python.org/2/library/logging.config.html#logging.config.fileConfig]
and [Python 3 docs|https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig].)
This can be fixed by adding {{disable_existing_loggers=False}} to the call of {{fileConfig}}.
> This has affected us at Clover Health because we use these database utility functions
in some of our tooling, and particularly our _tests_ of the tooling. Having all logging disabled
in the midst of our tests makes it more difficult to test our use of logging in completely
unrelated parts of our codebase.
> As an example, we were trying to use [pytest's caplog feature|https://docs.pytest.org/en/latest/logging.html#caplog-fixture],
but were unable to do so with logging globally disabled by {{fileConfig}}. Here's an example
of a test that fails with {{disable_existing_loggers=True}} (the default), but passes with
{{disable_existing_loggers=False}}.
> {code}
> import logging
> import pytest
> import airflow.utils.db as af_db
> LOGGER = logging.getLogger(__name__)
> @pytest.fixture(autouse=True)
> def resetdb():
>     af_db.resetdb()
> def test_caplog(caplog):
>     LOGGER.info('LINE 1')
>     assert caplog.record_tuples
>     assert 'LINE 1' in caplog.text
> {code}
> I'll submit a pull request shortly to add {{disable_existing_loggers=False}} to Airflow's
{{[env.py|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message