airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Maguire <andrewm4...@gmail.com>
Subject Re: Airflow 1.9 Logging Set Up
Date Mon, 23 Oct 2017 15:23:46 GMT
Cool,

So what's the best way for me to try this out - should i wait for it to be
merged to master and then upgrade from the relevant github source?

Cheers,
Andy

On Mon, Oct 23, 2017 at 3:45 PM Driesprong, Fokko <fokko@driesprong.frl>
wrote:

> Hi Andy,
>
> I've pushed a fix:
> https://github.com/apache/incubator-airflow/pull/2721/files
>
> I've tested locally, and this will add ${AIRFLOW_HOME}/config automatically
> to the pythonpath. Setting:
> logging_config_class = airflow_logging_settings.LOGGING_CONFIG
>
> And creating ${AIRFLOW_HOME}/config/airflow_logging_settings.py and
> providing a config as in the mail before, should work. I've tested it
> locally within a Python 3 docker container, and this fixed it. I also made
> some small changes to the config, based on your suggestions.
>
> Cheers, Fokko
>
>
>
>
> 2017-10-18 16:42 GMT+02:00 Andrew Maguire <andrewm4894@gmail.com>:
>
> > Cool - i'll help test once something ready,
> >
> > So is the expected approach (easiest for a normal users wanting to just
> log
> > to GCS) here for me to
> >
> > 1. copy the "airflow_local_settings.py" from here
> > <https://github.com/apache/incubator-airflow/blob/master/
> > airflow/config_templates/airflow_local_settings.py>
> > into {AIRFLOW_HOME}/plugins
> > 2. uncomment the GCS stuff, add a line to pick up GCS_LOG_FOLDER from
> > {AIRFLOW_HOME}/airflow.cfg
> > 3. add an __init__.py to {AIRFLOW_HOME}/plugins
> > 4. update the {AIRFLOW_HOME}/airflow.cfg accordingly
> >
> > If so, it seems like really i'm not doing anything of substance in steps
> > 1-3 but just potential places for less technical users (like me :) ) to
> > mess up.
> >
> > When i first set up airflow i was really impressed with how easy it was
> to
> > just figure out airflow.cfg and get things up and running.
> >
> > Maybe the plan is to cut down some of these steps once the new logging is
> > more final but just wanted to give a little bit of feedback - being able
> to
> > just set everything up via airflow.cfg is great.
> >
> > Cheers,
> > Andy
> >
> > On Wed, Oct 18, 2017 at 2:51 PM Driesprong, Fokko <fokko@driesprong.frl>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'll push a fix today. I've created a Jira ticket:
> > > https://issues.apache.org/jira/browse/AIRFLOW-1731
> > >
> > > Cheers, Fokko
> > >
> > > 2017-10-17 19:27 GMT+02:00 Driesprong, Fokko <fokko@driesprong.frl>:
> > >
> > > > Hi Andy,
> > > >
> > > > I see something weird. While debugging the code, the paths as
> > documented
> > > > on the updating.md are not on the pythonpath:
> > > > ['/usr/local/bin', '/usr/local/lib/python36.zip',
> > > > '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload',
> > > > '/usr/local/lib/python3.6/site-packages']
> > > >
> > > > This is an installed instance of 1.9-alpha using the .tar.gz of
> Chris.
> > I
> > > > would also expect more paths appended to the PYTHONPATH. So when I
> try
> > to
> > > > run an example with the config provided by Andrew:
> > > > logging_config_class = airflow_logging_settings.LOGGING_CONFIG
> > > >
> > > > I end up with the same error as Andrew:
> > > > root@9e3cf03c0544:~# airflow test example_bash_operator runme_0
> > > 2017-01-01
> > > > <20%2017%2001%2001>
> > > > Current path:
> > > > ['/usr/local/bin', '/usr/local/lib/python36.zip',
> > > > '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload',
> > > > '/usr/local/lib/python3.6/site-packages']
> > > > Traceback (most recent call last):
> > > >   File
> > > "/usr/local/lib/python3.6/site-packages/airflow/logging_config.py",
> > > > line 40, in configure_logging
> > > >     logging_config = import_string(logging_class_path)
> > > >   File
> > > "/usr/local/lib/python3.6/site-packages/airflow/utils/
> > module_loading.py",
> > > > line 28, in import_string
> > > >     module = import_module(module_path)
> > > >   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in
> > > > import_module
> > > >     return _bootstrap._gcd_import(name[level:], package, level)
> > > >   File "<frozen importlib._bootstrap>", line 978, in _gcd_import
> > > >   File "<frozen importlib._bootstrap>", line 961, in _find_and_load
> > > >   File "<frozen importlib._bootstrap>", line 948, in
> > > > _find_and_load_unlocked
> > > > ModuleNotFoundError: No module named 'airflow_logging_settings'
> > > >
> > > > During handling of the above exception, another exception occurred:
> > > >
> > > > Traceback (most recent call last):
> > > >   File "/usr/local/bin/airflow", line 16, in <module>
> > > >     from airflow import configuration
> > > >   File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py",
> > line
> > > > 31, in <module>
> > > >     from airflow import settings
> > > >   File "/usr/local/lib/python3.6/site-packages/airflow/settings.py",
> > line
> > > > 148, in <module>
> > > >     configure_logging()
> > > >   File
> > > "/usr/local/lib/python3.6/site-packages/airflow/logging_config.py",
> > > > line 52, in configure_logging
> > > >     'Unable to load custom logging from
> {}'.format(logging_class_path)
> > > > ImportError: Unable to load custom logging from
> > airflow_logging_settings.
> > > > LOGGING_CONFIG
> > > >
> > > >
> > > > When explicitly passing the path when starting the test:
> > > > root@9e3cf03c0544:~# PYTHONPATH=/root/airflow/plugins/ airflow test
> > > > example_bash_operator runme_0 2017-01-01 <20%2017%2001%2001>
> > > > Current path:
> > > > ['/usr/local/bin', '/root/airflow/plugins',
> > > '/usr/local/lib/python36.zip',
> > > > '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload',
> > > > '/usr/local/lib/python3.6/site-packages']
> > > > [2017-10-17 17:21:19,793] {__init__.py:45} INFO - Using executor
> > > > SequentialExecutor
> > > > [2017-10-17 17:21:19,835] {models.py:186} INFO - Filling up the
> DagBag
> > > > from /root/airflow/dags
> > > > [2017-10-17 17:21:19,880] {dag.py:31} WARNING - test warn
> > > > [2017-10-17 17:21:19,880] {dag.py:32} INFO - test info
> > > > [2017-10-17 17:21:19,902] {models.py:1165} INFO - Dependencies all
> met
> > > for
> > > > <TaskInstance: example_bash_operator.runme_0 2017-01-01 00:00:00
> > [None]>
> > > > [2017-10-17 17:21:19,904] {models.py:1165} INFO - Dependencies all
> met
> > > for
> > > > <TaskInstance: example_bash_operator.runme_0 2017-01-01 00:00:00
> > [None]>
> > > > [2017-10-17 17:21:19,905] {models.py:1375} INFO -
> > > > ------------------------------------------------------------
> > > > --------------------
> > > > Starting attempt 1 of 1
> > > > ------------------------------------------------------------
> > > > --------------------
> > > >
> > > > [2017-10-17 17:21:19,905] {models.py:1396} INFO - Executing
> > > > <Task(BashOperator): runme_0> on 2017-01-01 00:00:00
> > > > [2017-10-17 17:21:19,916] {bash_operator.py:70} INFO - Tmp dir root
> > > > location:
> > > >  /tmp
> > > > [2017-10-17 17:21:19,919] {bash_operator.py:80} INFO - Temporary
> script
> > > > location: /tmp/airflowtmpmco2abof//tmp/airflowtmpmco2abof/runme_
> > 0w53ja72x
> > > > [2017-10-17 17:21:19,919] {bash_operator.py:82} INFO - Running
> command:
> > > > echo "example_bash_operator__runme_0__20170101 <20%2017%2001%2001>"
> &&
> > > > sleep 1
> > > > [2017-10-17 17:21:19,924] {bash_operator.py:91} INFO - Output:
> > > > [2017-10-17 17:21:19,925] {bash_operator.py:95} INFO -
> > > > example_bash_operator__runme_0__20170101 <20%2017%2001%2001>
> > > > [2017-10-17 17:21:20,928] {bash_operator.py:99} INFO - Command exited
> > > with
> > > > return code 0
> > > >
> > > > This will require a patch for 1.9 because I expected more paths being
> > > > loaded by default. I checked this before, and the paths mentioned in
> > the
> > > > updating.md where present, anyone any idea?
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > >
> > > > 2017-10-17 13:04 GMT+02:00 Andrew Maguire <andrewm4894@gmail.com>:
> > > >
> > > >> Hey,
> > > >>
> > > >> My airflow_logging_settings.py is attached. (is the same as the
> > > >> airflow_local_settings.py you mention - i think i saw both and
> > > >> 'airflow_local_settings.py' and an 'airflow_logging_settings.py'
> > > referenced
> > > >> in two separate places although i think it's pretty much the same
> > thing,
> > > >> just slightly different filenames. I think one is mentioned in the
> > > >> updating.md file and the other is what the file is actually called
> in
> > > >> the github repo. Am guessing does not matter so long as i refrence
> the
> > > >> correct file in airflow.cfg).
> > > >>
> > > >> I had not renamed DEFAULT_LOGGING_CONFIG to LOGGING_CONFIG in my
> > > >> airflow_logging_settings.py. I did and uncommented the class line
in
> > my
> > > >> airflow.cfg and still got below error:
> > > >>
> > > >> andrew_maguire@airflow-server:~$ airflow list_dags
> > > >> Traceback (most recent call last):
> > > >>   File "/usr/local/bin/airflow", line 16, in <module>
> > > >>     from airflow import configuration
> > > >>   File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py",
> > > line 31, in <module>
> > > >>     from airflow import settings
> > > >>   File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py",
> > > line 148, in <module>
> > > >>     configure_logging()
> > > >>   File
> > > "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py",
> line
> > > 47, in configure_logging
> > > >>     'Unable to load custom logging from
> {}'.format(logging_class_path)
> > > >> ImportError: Unable to load custom logging from
> > > plugins.airflow_logging_settings.LOGGING_CONFIG
> > > >>
> > > >> So don't think my error even got that far.
> > > >>
> > > >> Happy to help debug or test or be of any use.
> > > >>
> > > >> Cheers,
> > > >> Andy
> > > >>
> > > >>
> > > >> On Tue, Oct 17, 2017 at 11:43 AM Driesprong, Fokko
> > <fokko@driesprong.frl
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi Andy,
> > > >>>
> > > >>> Thanks for reaching out. We are debugging the new logging, and
> input
> > > from
> > > >>> the community is highly appreciated.
> > > >>>
> > > >>> If you are using Python 2, you'll need to put an empty __init__.py
> in
> > > all
> > > >>> the directory, so ~/airflow/plugins/__init__.py, this needs to
be
> > > empty.
> > > >>> Could you share your airflow_local_settings.py? If there are any
> GCS
> > > >>> credentials, please remove them. Please check that you've renamed
> > > >>> the DEFAULT_LOGGING_CONFIG variable to LOGGING_CONFIG, this might
> not
> > > be
> > > >>> evident from the updating.md.
> > > >>>
> > > >>> Cheers, Fokko
> > > >>>
> > > >>> 2017-10-17 12:00 GMT+02:00 Andrew Maguire <andrewm4894@gmail.com>:
> > > >>>
> > > >>> > Hi,
> > > >>> >
> > > >>> > I've updated to 1.9 but am having trouble setting the
> > > >>> logging_config_class
> > > >>> > class path in the airflow.cfg file.
> > > >>> >
> > > >>> > Currently i have below files in {AIRFLOW_HOME}/plugins
> > > >>> >
> > > >>> > [image: image.png]
> > > >>> > Where airflow_logging_settings.py is just a copy of this
file
> > > >>> > <https://github.com/apache/incubator-airflow/blob/master/air
> > > >>> flow/config_templates/airflow_local_settings.py>
> > > >>>
> > > >>> > but with the GCS stuff uncommented and a line added for
> > > GCS_LOG_FOLDER
> > > >>> to
> > > >>> > be pulled from airflow.cfg just like BASE_LOG_FOLDER
> > > >>> >
> > > >>> > then in my {AIRFLOW_HOME}/airflow.cfg file i have this following
> > line
> > > >>> to
> > > >>> > set up the log stuff:
> > > >>> >
> > > >>> > # The folder where airflow should store its log files
> > > >>> > # This path must be absolute
> > > >>> > base_log_folder = {AIRFLOW_HOME}/logs
> > > >>> >
> > > >>> > gcs_log_folder = gs://pmc-airflow/logs
> > > >>> >
> > > >>> > # Airflow can store logs remotely in AWS S3 or Google Cloud
> > Storage.
> > > >>> Users
> > > >>> > # must supply an Airflow connection id that provides access
to
> the
> > > >>> storage
> > > >>> > # location.
> > > >>> > remote_log_conn_id = my_gcp_connection
> > > >>> > encrypt_s3_logs = False
> > > >>> >
> > > >>> > # Logging level
> > > >>> > logging_level = INFO
> > > >>> >
> > > >>> > # Logging class
> > > >>> > # Specify the class that will specify the logging configuration
> > > >>> > # This class has to be on the python classpath
> > > >>> > logging_config_class =
> > > plugins.airflow_logging_settings.LOGGING_CONFIG
> > > >>> >
> > > >>> > # Log format
> > > >>> > log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}}
> > > >>> %%(levelname)s - %%(message)s
> > > >>> > simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
> > > >>> >
> > > >>> >
> > > >>> > However if i run "airflow list_dags" i now get:
> > > >>> >
> > > >>> > andrew_maguire@airflow-server:~/airflow$ airflow list_dags
> > > >>> > Traceback (most recent call last):
> > > >>> >   File "/usr/local/bin/airflow", line 16, in <module>
> > > >>> >     from airflow import configuration
> > > >>> >   File "/usr/local/lib/python2.7/dist-packages/airflow/__init__
> > .py",
> > > >>> line 31, in <module>
> > > >>> >     from airflow import settings
> > > >>> >   File "/usr/local/lib/python2.7/dist-packages/airflow/
> > settings.py",
> > > >>> line 148, in <module>
> > > >>> >     configure_logging()
> > > >>> >   File
> > > "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py",
> > > >>> line 47, in configure_logging
> > > >>> >     'Unable to load custom logging from
> > > {}'.format(logging_class_path)
> > > >>> > ImportError: Unable to load custom logging from
> > > >>> plugins.airflow_logging_settings.LOGGING_CONFIG
> > > >>> >
> > > >>> > If i go back into my airflow.cfg and comment out the line:
> > > >>> >
> > > >>> > logging_config_class =
> > > plugins.airflow_logging_settings.LOGGING_CONFIG
> > > >>> >
> > > >>> > Things work again but i'm only doing local logging.
> > > >>> >
> > > >>> > So am sure i'm doing something wrong here in how i'm setting
that
> > > line
> > > >>> in the airflow.cfg file.
> > > >>> >
> > > >>> > So what i did was.
> > > >>> >
> > > >>> > 1. create a folder {AIRFLOW_HOME}/plugins - i did this as
> > > updating.md
> > > >>> <
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
> > >
> > > >>> mentioned that "The logging configuration file that contains the
> > > >>> configuration needs te on the the PYTHONPATH, for example in
> > > ~/airflow/dags
> > > >>> or ~/airflow/plugins. These directories are loaded by default".
> > > >>> >
> > > >>> > 2. copy this file <https://github.com/apache/inc
> > > >>> ubator-airflow/blob/master/airflow/config_templates/airflow_
> > > >>> local_settings.py> into {AIRFLOW_HOME}/plugins with the GCS
> changes i
> > > >>> mentioned.
> > > >>> >
> > > >>> > 3. create an __init__.py file in {AIRFLOW_HOME}/plugins -
do i
> need
> > > to
> > > >>> put anyting in partiucular in here?
> > > >>> >
> > > >>> > 4. update  {AIRFLOW_HOME}/airflow.cfg as above.
> > > >>> >
> > > >>> > Can someone help me figure out where i went wrong?
> > > >>> >
> > > >>> > I'm hesitant to change or add anytihng to the pythonpath
as an
> not
> > > >>> 100% sure what i'm doing. So was hoping to just drop the logging
> > config
> > > >>> file somewhere it would automatically be picked up. And i'm also
> not
> > > really
> > > >>> sure about python packages and class paths etc so kinda feeling
my
> > way
> > > >>> through it but not confident.
> > > >>> >
> > > >>> > Cheers
> > > >>> >
> > > >>> > Andy
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>>
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message