airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Davis (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-2127) Airflow's Alembic migrations globally disable logging
Date Tue, 20 Feb 2018 19:21:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matt Davis updated AIRFLOW-2127:
--------------------------------
    Description: 
When running Airflow's {{[upgradedb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L295]}},
{{[resetdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L311]}},
and {{[initdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L83]}} functions
logging is disabled thereafter for all but the {{sqlalchemy.engine}} and {{alembic}} loggers.
This is caused [this usage|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]
of Python's {{fileConfig}} function, which by default disables all loggers that aren't part
of the supplied configuration. (See [Python 2 docs|https://docs.python.org/2/library/logging.config.html#logging.config.fileConfig]
and [Python 3 docs|https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig].)
This can be fixed by adding {{disable_existing_loggers=False}} to the call of {{fileConfig}}.

This has affected us at Clover Health because we use these database utility functions in some
of our tooling, and particularly our _tests_ of the tooling. Having all logging disabled
in the midst of our tests makes it more difficult to test our use of logging in completely
unrelated parts of our codebase.

As an example, we were trying to use [pytest's caplog feature|https://docs.pytest.org/en/latest/logging.html#caplog-fixture],
but were unable to do so with logging globally disabled by {{fileConfig}}. Here's an example
of a test that fails with {{disable_existing_loggers=True}} (the default), but passes with
{{disable_existing_loggers=False}}.
{code}
import logging

import pytest

import airflow.utils.db as af_db

LOGGER = logging.getLogger(__name__)

@pytest.fixture(autouse=True)
def resetdb():
    af_db.resetdb()

def test_caplog(caplog):
    LOGGER.info('LINE 1')
    assert caplog.record_tuples
    assert 'LINE 1' in caplog.text
{code}
I'll submit a pull request shortly to add {{disable_existing_loggers=False}} to Airflow's
{{[env.py|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]}}.

  was:
When running Airflow's {{[upgradedb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L295]}},
{{[resetdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L311]}},
and {{[initdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L83]}} functions
logging is disabled thereafter for all but the {{sqlalchemy.engine}} and {{alembic}} loggers.
This is caused [this usage|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]
of Python's {{fileConfig}} function, which by default disables all loggers that aren't part
of the supplied configuration. (See [Python 2 docs|https://docs.python.org/2/library/logging.config.html#logging.config.fileConfig]
and [Python 3 docs|https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig].)
This can be fixed by adding {{disable_existing_loggers=False}} to the call of {{fileConfig}}.

This has affected us at Clover Health because we use these database utility functions in some
of our tooling, and particularly our _tests_ of the tooling. Having all logging disabled
in the midst of our tests has it difficult to test our use of logging in completely unrelated
parts of our codebase.

As an example, we were trying to use [pytest's caplog feature|https://docs.pytest.org/en/latest/logging.html#caplog-fixture],
but were unable to do so with logging globally disabled by {{fileConfig}}. Here's an example
of a test that fails with {{disable_existing_loggers=True}} (the default), but passes with
{{disable_existing_loggers=False}}.
{code}
import logging

import pytest

import airflow.utils.db as af_db

LOGGER = logging.getLogger(__name__)

@pytest.fixture(autouse=True)
def resetdb():
    af_db.resetdb()

def test_caplog(caplog):
    LOGGER.info('LINE 1')
    assert caplog.record_tuples
    assert 'LINE 1' in caplog.text
{code}
I'll submit a pull request shortly to add {{disable_existing_loggers=False}} to Airflow's
{{[env.py|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]}}.


> Airflow's Alembic migrations globally disable logging
> -----------------------------------------------------
>
>                 Key: AIRFLOW-2127
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2127
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db
>            Reporter: Matt Davis
>            Priority: Major
>
> When running Airflow's {{[upgradedb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L295]}},
{{[resetdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L311]}},
and {{[initdb|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/utils/db.py#L83]}} functions
logging is disabled thereafter for all but the {{sqlalchemy.engine}} and {{alembic}} loggers.
This is caused [this usage|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]
of Python's {{fileConfig}} function, which by default disables all loggers that aren't part
of the supplied configuration. (See [Python 2 docs|https://docs.python.org/2/library/logging.config.html#logging.config.fileConfig]
and [Python 3 docs|https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig].)
This can be fixed by adding {{disable_existing_loggers=False}} to the call of {{fileConfig}}.
> This has affected us at Clover Health because we use these database utility functions
in some of our tooling, and particularly our _tests_ of the tooling. Having all logging disabled
in the midst of our tests makes it more difficult to test our use of logging in completely
unrelated parts of our codebase.
> As an example, we were trying to use [pytest's caplog feature|https://docs.pytest.org/en/latest/logging.html#caplog-fixture],
but were unable to do so with logging globally disabled by {{fileConfig}}. Here's an example
of a test that fails with {{disable_existing_loggers=True}} (the default), but passes with
{{disable_existing_loggers=False}}.
> {code}
> import logging
> import pytest
> import airflow.utils.db as af_db
> LOGGER = logging.getLogger(__name__)
> @pytest.fixture(autouse=True)
> def resetdb():
>     af_db.resetdb()
> def test_caplog(caplog):
>     LOGGER.info('LINE 1')
>     assert caplog.record_tuples
>     assert 'LINE 1' in caplog.text
> {code}
> I'll submit a pull request shortly to add {{disable_existing_loggers=False}} to Airflow's
{{[env.py|https://github.com/apache/incubator-airflow/blob/fc26cade87e181f162ecc8391ae16dccbe6f29c4/airflow/migrations/env.py#L28]}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message