airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: Airflow 1.9 Logging Set Up
Date Tue, 17 Oct 2017 17:27:05 GMT
Hi Andy,

I see something weird. While debugging the code, the paths as documented on
the updating.md are not on the pythonpath:
['/usr/local/bin', '/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages']

This is an installed instance of 1.9-alpha using the .tar.gz of Chris. I
would also expect more paths appended to the PYTHONPATH. So when I try to
run an example with the config provided by Andrew:
logging_config_class = airflow_logging_settings.LOGGING_CONFIG

I end up with the same error as Andrew:
root@9e3cf03c0544:~# airflow test example_bash_operator runme_0 2017-01-01
Current path:
['/usr/local/bin', '/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages']
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/logging_config.py",
line 40, in configure_logging
    logging_config = import_string(logging_class_path)
  File
"/usr/local/lib/python3.6/site-packages/airflow/utils/module_loading.py",
line 28, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in
import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow_logging_settings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 16, in <module>
    from airflow import configuration
  File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line
31, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line
148, in <module>
    configure_logging()
  File "/usr/local/lib/python3.6/site-packages/airflow/logging_config.py",
line 52, in configure_logging
    'Unable to load custom logging from {}'.format(logging_class_path)
ImportError: Unable to load custom logging from
airflow_logging_settings.LOGGING_CONFIG


When explicitly passing the path when starting the test:
root@9e3cf03c0544:~# PYTHONPATH=/root/airflow/plugins/ airflow test
example_bash_operator runme_0 2017-01-01
Current path:
['/usr/local/bin', '/root/airflow/plugins', '/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages']
[2017-10-17 17:21:19,793] {__init__.py:45} INFO - Using executor
SequentialExecutor
[2017-10-17 17:21:19,835] {models.py:186} INFO - Filling up the DagBag from
/root/airflow/dags
[2017-10-17 17:21:19,880] {dag.py:31} WARNING - test warn
[2017-10-17 17:21:19,880] {dag.py:32} INFO - test info
[2017-10-17 17:21:19,902] {models.py:1165} INFO - Dependencies all met for
<TaskInstance: example_bash_operator.runme_0 2017-01-01 00:00:00 [None]>
[2017-10-17 17:21:19,904] {models.py:1165} INFO - Dependencies all met for
<TaskInstance: example_bash_operator.runme_0 2017-01-01 00:00:00 [None]>
[2017-10-17 17:21:19,905] {models.py:1375} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2017-10-17 17:21:19,905] {models.py:1396} INFO - Executing
<Task(BashOperator): runme_0> on 2017-01-01 00:00:00
[2017-10-17 17:21:19,916] {bash_operator.py:70} INFO - Tmp dir root
location:
 /tmp
[2017-10-17 17:21:19,919] {bash_operator.py:80} INFO - Temporary script
location: /tmp/airflowtmpmco2abof//tmp/airflowtmpmco2abof/runme_0w53ja72x
[2017-10-17 17:21:19,919] {bash_operator.py:82} INFO - Running command:
echo "example_bash_operator__runme_0__20170101" && sleep 1
[2017-10-17 17:21:19,924] {bash_operator.py:91} INFO - Output:
[2017-10-17 17:21:19,925] {bash_operator.py:95} INFO -
example_bash_operator__runme_0__20170101
[2017-10-17 17:21:20,928] {bash_operator.py:99} INFO - Command exited with
return code 0

This will require a patch for 1.9 because I expected more paths being
loaded by default. I checked this before, and the paths mentioned in the
updating.md where present, anyone any idea?

Cheers, Fokko



2017-10-17 13:04 GMT+02:00 Andrew Maguire <andrewm4894@gmail.com>:

> Hey,
>
> My airflow_logging_settings.py is attached. (is the same as the
> airflow_local_settings.py you mention - i think i saw both and
> 'airflow_local_settings.py' and an 'airflow_logging_settings.py' referenced
> in two separate places although i think it's pretty much the same thing,
> just slightly different filenames. I think one is mentioned in the
> updating.md file and the other is what the file is actually called in the
> github repo. Am guessing does not matter so long as i refrence the correct
> file in airflow.cfg).
>
> I had not renamed DEFAULT_LOGGING_CONFIG to LOGGING_CONFIG in my
> airflow_logging_settings.py. I did and uncommented the class line in my
> airflow.cfg and still got below error:
>
> andrew_maguire@airflow-server:~$ airflow list_dags
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 16, in <module>
>     from airflow import configuration
>   File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 31, in <module>
>     from airflow import settings
>   File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 148, in <module>
>     configure_logging()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py", line 47, in
configure_logging
>     'Unable to load custom logging from {}'.format(logging_class_path)
> ImportError: Unable to load custom logging from plugins.airflow_logging_settings.LOGGING_CONFIG
>
> So don't think my error even got that far.
>
> Happy to help debug or test or be of any use.
>
> Cheers,
> Andy
>
>
> On Tue, Oct 17, 2017 at 11:43 AM Driesprong, Fokko <fokko@driesprong.frl>
> wrote:
>
>> Hi Andy,
>>
>> Thanks for reaching out. We are debugging the new logging, and input from
>> the community is highly appreciated.
>>
>> If you are using Python 2, you'll need to put an empty __init__.py in all
>> the directory, so ~/airflow/plugins/__init__.py, this needs to be empty.
>> Could you share your airflow_local_settings.py? If there are any GCS
>> credentials, please remove them. Please check that you've renamed
>> the DEFAULT_LOGGING_CONFIG variable to LOGGING_CONFIG, this might not be
>> evident from the updating.md.
>>
>> Cheers, Fokko
>>
>> 2017-10-17 12:00 GMT+02:00 Andrew Maguire <andrewm4894@gmail.com>:
>>
>> > Hi,
>> >
>> > I've updated to 1.9 but am having trouble setting the
>> logging_config_class
>> > class path in the airflow.cfg file.
>> >
>> > Currently i have below files in {AIRFLOW_HOME}/plugins
>> >
>> > [image: image.png]
>> > Where airflow_logging_settings.py is just a copy of this file
>> > <https://github.com/apache/incubator-airflow/blob/master/
>> airflow/config_templates/airflow_local_settings.py>
>>
>> > but with the GCS stuff uncommented and a line added for GCS_LOG_FOLDER
>> to
>> > be pulled from airflow.cfg just like BASE_LOG_FOLDER
>> >
>> > then in my {AIRFLOW_HOME}/airflow.cfg file i have this following line to
>> > set up the log stuff:
>> >
>> > # The folder where airflow should store its log files
>> > # This path must be absolute
>> > base_log_folder = {AIRFLOW_HOME}/logs
>> >
>> > gcs_log_folder = gs://pmc-airflow/logs
>> >
>> > # Airflow can store logs remotely in AWS S3 or Google Cloud Storage.
>> Users
>> > # must supply an Airflow connection id that provides access to the
>> storage
>> > # location.
>> > remote_log_conn_id = my_gcp_connection
>> > encrypt_s3_logs = False
>> >
>> > # Logging level
>> > logging_level = INFO
>> >
>> > # Logging class
>> > # Specify the class that will specify the logging configuration
>> > # This class has to be on the python classpath
>> > logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG
>> >
>> > # Log format
>> > log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}}
>> %%(levelname)s - %%(message)s
>> > simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
>> >
>> >
>> > However if i run "airflow list_dags" i now get:
>> >
>> > andrew_maguire@airflow-server:~/airflow$ airflow list_dags
>> > Traceback (most recent call last):
>> >   File "/usr/local/bin/airflow", line 16, in <module>
>> >     from airflow import configuration
>> >   File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py",
>> line 31, in <module>
>> >     from airflow import settings
>> >   File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py",
>> line 148, in <module>
>> >     configure_logging()
>> >   File "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py",
>> line 47, in configure_logging
>> >     'Unable to load custom logging from {}'.format(logging_class_path)
>> > ImportError: Unable to load custom logging from plugins.airflow_logging_
>> settings.LOGGING_CONFIG
>> >
>> > If i go back into my airflow.cfg and comment out the line:
>> >
>> > logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG
>> >
>> > Things work again but i'm only doing local logging.
>> >
>> > So am sure i'm doing something wrong here in how i'm setting that line
>> in the airflow.cfg file.
>> >
>> > So what i did was.
>> >
>> > 1. create a folder {AIRFLOW_HOME}/plugins - i did this as updating.md <
>> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md>
>> mentioned that "The logging configuration file that contains the
>> configuration needs te on the the PYTHONPATH, for example in ~/airflow/dags
>> or ~/airflow/plugins. These directories are loaded by default".
>> >
>> > 2. copy this file <https://github.com/apache/
>> incubator-airflow/blob/master/airflow/config_templates/
>> airflow_local_settings.py> into {AIRFLOW_HOME}/plugins with the GCS
>> changes i mentioned.
>> >
>> > 3. create an __init__.py file in {AIRFLOW_HOME}/plugins - do i need to
>> put anyting in partiucular in here?
>> >
>> > 4. update  {AIRFLOW_HOME}/airflow.cfg as above.
>> >
>> > Can someone help me figure out where i went wrong?
>> >
>> > I'm hesitant to change or add anytihng to the pythonpath as an not 100%
>> sure what i'm doing. So was hoping to just drop the logging config file
>> somewhere it would automatically be picked up. And i'm also not really sure
>> about python packages and class paths etc so kinda feeling my way through
>> it but not confident.
>> >
>> > Cheers
>> >
>> > Andy
>> >
>> >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message