airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: How to bes use Google Cloud Storage for logging?
Date Wed, 20 Dec 2017 06:47:03 GMT
You seem to mix master and 1-9-stable. The RedirectStdHandler is only available in master.

B.

Verstuurd vanaf mijn iPad

> Op 19 dec. 2017 om 23:37 heeft Kevin Lam <kevin@fathomhealth.co> het volgende geschreven:
> 
> Hi Feng,
> 
> Thanks for your help! Got it, will try to push on the python based logging
> config.
> 
> I'm trying to set-up the GCS logging on airflow v1-9-stable and my
> logging_config.py seems to be causing a python import error, caused by
> 'from airflow import configuration'
> 
> "Initialize database...
> Unable to load the config, contains a configuration error.
> Traceback (most recent call last):
>  File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
>    self.importer(used)
> ImportError: No module named
> 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> 'airflow.utils.log.logging_mixin' is not a package
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>  File "/usr/lib/python3.5/logging/config.py", line 558, in configure
>    handler = self.configure_handler(handlers[name])
>  File "/usr/lib/python3.5/logging/config.py", line 708, in
> configure_handler
>    klass = self.resolve(cname)
>  File "/usr/lib/python3.5/logging/config.py", line 391, in resolve
>    raise v
>  File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
>    self.importer(used)
> ValueError: Cannot resolve
> 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
> 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> 'airflow.utils.log.logging_mixin' is not a package
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>  File "/usr/local/bin/airflow", line 16, in <module>
>    from airflow import configuration
>  File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", line
> 31, in <module>
>    from airflow import settings
>  File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py", line
> 148, in <module>
>    configure_logging()
>  File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
> line 75, in configure_logging
>    raise e
>  File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
> line 70, in configure_logging
>    dictConfig(logging_config)
>  File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
>    dictConfigClass(config).configure()
>  File "/usr/lib/python3.5/logging/config.py", line 566, in configure
>    '%r: %s' % (name, e))
> ValueError: Unable to configure handler 'console': Cannot resolve
> 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
> 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> 'airflow.utils.log.logging_mixin' is not a package
> HTTP/1.1 200 OK
> Unable to load the config, contains a configuration error.
> Traceback (most recent call last):
>  File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
>    self.importer(used)
> ImportError: No module named
> 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> 'airflow.utils.log.logging_mixin' is not a package
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>  File "/usr/lib/python3.5/logging/config.py", line 558, in configure
>    handler = self.configure_handler(handlers[name])
>  File "/usr/lib/python3.5/logging/config.py", line 708, in
> configure_handler
>    klass = self.resolve(cname)
>  File "/usr/lib/python3.5/logging/config.py", line 391, in resolve
>    raise v
>  File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
>    self.importer(used)
> ValueError: Cannot resolve
> 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
> 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> 'airflow.utils.log.logging_mixin' is not a package
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>  File "/usr/local/bin/airflow", line 16, in <module>
>    from airflow import configuration
>  File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py", line
> 31, in <module>
>    from airflow import settings
>  File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py", line
> 148, in <module>
>    configure_logging()
>  File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
> line 75, in configure_logging
>    raise e
>  File "/usr/local/lib/python3.5/dist-packages/airflow/logging_config.py",
> line 70, in configure_logging
>    dictConfig(logging_config)
>  File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
>    dictConfigClass(config).configure()
>  File "/usr/lib/python3.5/logging/config.py", line 566, in configure
>    '%r: %s' % (name, e))
> ValueError: Unable to configure handler 'console': Cannot resolve
> 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
> 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> 'airflow.utils.log.logging_mixin' is not a package"
> 
> Have you encountered this before?
> 
>> On Mon, Dec 18, 2017 at 8:53 PM, Feng Lu <fenglu@google.com.invalid> wrote:
>> 
>> Hi Kevin,
>> 
>> Kindly see my reply inline:
>> 
>>> On Mon, Dec 18, 2017 at 3:28 PM, Kevin Lam <kevin@fathomhealth.co> wrote:
>>> 
>>> Hi,
>>> 
>>> I'm trying to get airflow to use GCS for logging purposes and had a few
>>> questions.
>>> 
>>> We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow
>>> deployment (similar to https://github.com/mumoshu/kube-airflow)
>>> 
>>> 1/ Seems like the logging code has been going through some changes in the
>>> recent versions. What's the correct way to set up GCS for logging? Is it
>> by
>>> just specifying remote_base_log_folder and remote_log_conn_id in
>>> airflow.cfg? Or by following this guide:
>>> http://airflow.readthedocs.io/en/latest/integration.html#gcp, using the
>>> python based logging config? Is there an Airflow version that we should
>> use
>>> to be most stable?
>>> 
>> The python based logging config is the right place to make changes, in our
>> test setup, we override the airflow_local_settings.py similarly to the link
>> you pasted.
>> You may also want to config: [core]task_log_reader = gcs.task
>> 
>> 
>>> 
>>> 2/ Is there a way to encode the connection for GCS in a file so that one
>>> doesn't have to open the webserver and create it from the admin panel?
>> It'd
>>> be nice if the GCS connection would be automatically created.
>>> 
>> Unfortunately GCS connection ties to some GCP project and is impossible to
>> pre-populate.
>> Airflow1.9 should fix the gcp connection type issue  (
>> https://github.com/apache/incubator-airflow/commit/
>> 2f107d8a30910fd025774004d5c4c95407ed55c5),
>> so you can use airflow connections CLI directly.
>> 
>> 
>>> 
>>> Thanks in advance for your help!
>>> 
>> 

Mime
View raw message