airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Lu <fen...@google.com.INVALID>
Subject Re: How to bes use Google Cloud Storage for logging?
Date Wed, 20 Dec 2017 01:10:41 GMT
We are using v1.9rc6 and it seems to work fine with directly editing
the airflow/config_templates/airflow_local_settings.py

On Tue, Dec 19, 2017 at 2:44 PM, Kevin Lam <kevin@fathomhealth.co> wrote:

> Looks like it might be related to
> https://github.com/apache/incubator-airflow/commit/
> 02ff8ae35dd16e6f23d29d7b24a5fb9c09d0b7a4?
> Why isn't this fix on the v1-9 branches? Should I be using master instead?
>
> On Tue, Dec 19, 2017 at 5:37 PM, Kevin Lam <kevin@fathomhealth.co> wrote:
>
> > Hi Feng,
> >
> > Thanks for your help! Got it, will try to push on the python based
> logging
> > config.
> >
> > I'm trying to set-up the GCS logging on airflow v1-9-stable and my
> > logging_config.py seems to be causing a python import error, caused by
> > 'from airflow import configuration'
> >
> > "Initialize database...
> > Unable to load the config, contains a configuration error.
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
> >     self.importer(used)
> > ImportError: No module named 'airflow.utils.log.logging_
> mixin.RedirectStdHandler';
> > 'airflow.utils.log.logging_mixin' is not a package
> >
> > The above exception was the direct cause of the following exception:
> >
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.5/logging/config.py", line 558, in configure
> >     handler = self.configure_handler(handlers[name])
> >   File "/usr/lib/python3.5/logging/config.py", line 708, in
> > configure_handler
> >     klass = self.resolve(cname)
> >   File "/usr/lib/python3.5/logging/config.py", line 391, in resolve
> >     raise v
> >   File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
> >     self.importer(used)
> > ValueError: Cannot resolve 'airflow.utils.log.logging_
> mixin.RedirectStdHandler':
> > No module named 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> > 'airflow.utils.log.logging_mixin' is not a package
> >
> > During handling of the above exception, another exception occurred:
> >
> > Traceback (most recent call last):
> >   File "/usr/local/bin/airflow", line 16, in <module>
> >     from airflow import configuration
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py",
> line
> > 31, in <module>
> >     from airflow import settings
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py",
> line
> > 148, in <module>
> >     configure_logging()
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/logging_
> config.py",
> > line 75, in configure_logging
> >     raise e
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/logging_
> config.py",
> > line 70, in configure_logging
> >     dictConfig(logging_config)
> >   File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
> >     dictConfigClass(config).configure()
> >   File "/usr/lib/python3.5/logging/config.py", line 566, in configure
> >     '%r: %s' % (name, e))
> > ValueError: Unable to configure handler 'console': Cannot resolve
> > 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
> > 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> > 'airflow.utils.log.logging_mixin' is not a package
> > HTTP/1.1 200 OK
> > Unable to load the config, contains a configuration error.
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
> >     self.importer(used)
> > ImportError: No module named 'airflow.utils.log.logging_
> mixin.RedirectStdHandler';
> > 'airflow.utils.log.logging_mixin' is not a package
> >
> > The above exception was the direct cause of the following exception:
> >
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.5/logging/config.py", line 558, in configure
> >     handler = self.configure_handler(handlers[name])
> >   File "/usr/lib/python3.5/logging/config.py", line 708, in
> > configure_handler
> >     klass = self.resolve(cname)
> >   File "/usr/lib/python3.5/logging/config.py", line 391, in resolve
> >     raise v
> >   File "/usr/lib/python3.5/logging/config.py", line 384, in resolve
> >     self.importer(used)
> > ValueError: Cannot resolve 'airflow.utils.log.logging_
> mixin.RedirectStdHandler':
> > No module named 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> > 'airflow.utils.log.logging_mixin' is not a package
> >
> > During handling of the above exception, another exception occurred:
> >
> > Traceback (most recent call last):
> >   File "/usr/local/bin/airflow", line 16, in <module>
> >     from airflow import configuration
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/__init__.py",
> line
> > 31, in <module>
> >     from airflow import settings
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/settings.py",
> line
> > 148, in <module>
> >     configure_logging()
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/logging_
> config.py",
> > line 75, in configure_logging
> >     raise e
> >   File "/usr/local/lib/python3.5/dist-packages/airflow/logging_
> config.py",
> > line 70, in configure_logging
> >     dictConfig(logging_config)
> >   File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
> >     dictConfigClass(config).configure()
> >   File "/usr/lib/python3.5/logging/config.py", line 566, in configure
> >     '%r: %s' % (name, e))
> > ValueError: Unable to configure handler 'console': Cannot resolve
> > 'airflow.utils.log.logging_mixin.RedirectStdHandler': No module named
> > 'airflow.utils.log.logging_mixin.RedirectStdHandler';
> > 'airflow.utils.log.logging_mixin' is not a package"
> >
> > Have you encountered this before?
> >
> > On Mon, Dec 18, 2017 at 8:53 PM, Feng Lu <fenglu@google.com.invalid>
> > wrote:
> >
> >> Hi Kevin,
> >>
> >> Kindly see my reply inline:
> >>
> >> On Mon, Dec 18, 2017 at 3:28 PM, Kevin Lam <kevin@fathomhealth.co>
> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm trying to get airflow to use GCS for logging purposes and had a
> few
> >> > questions.
> >> >
> >> > We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow
> >> > deployment (similar to https://github.com/mumoshu/kube-airflow)
> >> >
> >> > 1/ Seems like the logging code has been going through some changes in
> >> the
> >> > recent versions. What's the correct way to set up GCS for logging? Is
> >> it by
> >> > just specifying remote_base_log_folder and remote_log_conn_id in
> >> > airflow.cfg? Or by following this guide:
> >> > http://airflow.readthedocs.io/en/latest/integration.html#gcp, using
> the
> >> > python based logging config? Is there an Airflow version that we
> should
> >> use
> >> > to be most stable?
> >> >
> >> The python based logging config is the right place to make changes, in
> our
> >> test setup, we override the airflow_local_settings.py similarly to the
> >> link
> >> you pasted.
> >> You may also want to config: [core]task_log_reader = gcs.task
> >>
> >>
> >> >
> >> > 2/ Is there a way to encode the connection for GCS in a file so that
> one
> >> > doesn't have to open the webserver and create it from the admin panel?
> >> It'd
> >> > be nice if the GCS connection would be automatically created.
> >> >
> >> Unfortunately GCS connection ties to some GCP project and is impossible
> to
> >> pre-populate.
> >> Airflow1.9 should fix the gcp connection type issue  (
> >> https://github.com/apache/incubator-airflow/commit/2f107d8a3
> >> 0910fd025774004d5c4c95407ed55c5),
> >> so you can use airflow connections CLI directly.
> >>
> >>
> >> >
> >> > Thanks in advance for your help!
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message