airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Maguire <andrewm4...@gmail.com>
Subject Airflow 1.9 Logging Set Up
Date Tue, 17 Oct 2017 10:00:47 GMT
Hi,

I've updated to 1.9 but am having trouble setting the logging_config_class
class path in the airflow.cfg file.

Currently i have below files in {AIRFLOW_HOME}/plugins

[image: image.png]
Where airflow_logging_settings.py is just a copy of this file
<https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py>
but with the GCS stuff uncommented and a line added for GCS_LOG_FOLDER to
be pulled from airflow.cfg just like BASE_LOG_FOLDER

then in my {AIRFLOW_HOME}/airflow.cfg file i have this following line to
set up the log stuff:

# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = {AIRFLOW_HOME}/logs

gcs_log_folder = gs://pmc-airflow/logs

# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
# must supply an Airflow connection id that provides access to the storage
# location.
remote_log_conn_id = my_gcp_connection
encrypt_s3_logs = False

# Logging level
logging_level = INFO

# Logging class
# Specify the class that will specify the logging configuration
# This class has to be on the python classpath
logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG

# Log format
log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}}
%%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s


However if i run "airflow list_dags" i now get:

andrew_maguire@airflow-server:~/airflow$ airflow list_dags
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 16, in <module>
    from airflow import configuration
  File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py",
line 31, in <module>
    from airflow import settings
  File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py",
line 148, in <module>
    configure_logging()
  File "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py",
line 47, in configure_logging
    'Unable to load custom logging from {}'.format(logging_class_path)
ImportError: Unable to load custom logging from
plugins.airflow_logging_settings.LOGGING_CONFIG

If i go back into my airflow.cfg and comment out the line:

logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG

Things work again but i'm only doing local logging.

So am sure i'm doing something wrong here in how i'm setting that line
in the airflow.cfg file.

So what i did was.

1. create a folder {AIRFLOW_HOME}/plugins - i did this as updating.md
<https://github.com/apache/incubator-airflow/blob/master/UPDATING.md>
mentioned that "The logging configuration file that contains the
configuration needs te on the the PYTHONPATH, for example in
~/airflow/dags or ~/airflow/plugins. These directories are loaded by
default".

2. copy this file
<https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py>
into {AIRFLOW_HOME}/plugins with the GCS changes i mentioned.

3. create an __init__.py file in {AIRFLOW_HOME}/plugins - do i need to
put anyting in partiucular in here?

4. update  {AIRFLOW_HOME}/airflow.cfg as above.

Can someone help me figure out where i went wrong?

I'm hesitant to change or add anytihng to the pythonpath as an not
100% sure what i'm doing. So was hoping to just drop the logging
config file somewhere it would automatically be picked up. And i'm
also not really sure about python packages and class paths etc so
kinda feeling my way through it but not confident.

Cheers

Andy

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message