airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Velthuis (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (AIRFLOW-3279) Documentation for Google Logging unclear
Date Mon, 05 Nov 2018 09:49:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paul Velthuis reassigned AIRFLOW-3279:
--------------------------------------

       Assignee:     (was: Fokko Driesprong)
       Priority: Blocker  (was: Major)
    Description: 
The documentation of how to install logging to a Google Cloud bucket is unclear.

I am now following the tutorial on the airflow page:

[https://airflow.apache.org/howto/write-logs.html]

Here I find it unclear what part of the 'logger' I have to adjust in the `{{airflow/config_templates/airflow_local_settings.py}}`.

 

The adjustment states:

 
 # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of 'file.task'.
'loggers':
 Unknown macro: \{ 'airflow.task'}

 

However what I find in the template is:
|'loggers': \{\| \|'airflow.processor': { \\\\\\\\| \\\\\\\\|'handlers': ['processor'], \\\\\\\\|
\\\\\\\\|'level': LOG_LEVEL, \\\\\\\\| \\\\\\\\|'propagate': False, \\\\\\\\| \\\\\\\\|},|
|'airflow.task': { 
\| 
\|'handlers': ['task'], 
\| 
\|'level': LOG_LEVEL, 
\| 
\|'propagate': False, 
\| 
\|},|
|'flask_appbuilder': { 
\| 
\|'handler': ['console'], 
\| 
\|'level': FAB_LOG_LEVEL, 
\| 
\|'propagate': True, 
\| 
\|}|

},

 

Since for me it is very important to do it right at the first time I hope some clarity can
be provided in what has to be adjusted in the logger. Is it only the 'airflow.task' or more?

Furthermore, at step 6 it is a little unclear what remote_log_conn_id means. I would propose
to add a little more information to make this more clear.

 

The current error I am facing is:

Traceback (most recent call last):
 File "/usr/local/bin/airflow", line 16, in <module>
 from airflow import configuration
 File "/usr/local/lib/python2.7/site-packages/airflow/__init__.py", line 31, in <module>
 from airflow import settings
 File "/usr/local/lib/python2.7/site-packages/airflow/settings.py", line 198, in <module>
 configure_logging()
 File "/usr/local/lib/python2.7/site-packages/airflow/logging_config.py", line 71, in configure_logging
 dictConfig(logging_config)
 File "/usr/local/lib/python2.7/logging/config.py", line 794, in dictConfig
 dictConfigClass(config).configure()
 File "/usr/local/lib/python2.7/logging/config.py", line 568, in configure
 handler = self.configure_handler(handlers[name])
 File "/usr/local/lib/python2.7/logging/config.py", line 733, in configure_handler
 result = factory(**kwargs)
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py", line
30, in __init__
 super(GCSTaskHandler, self).__init__(base_log_folder, filename_template)
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/file_task_handler.py", line
46, in __init__
 self.filename_jinja_template = Template(self.filename_template)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 926, in __new__
 return env.from_string(source, template_class=cls)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 862, in from_string
 return cls.from_code(self, self.compile(source), globals, None)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 565, in compile
 self.handle_exception(exc_info, source_hint=source_hint)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
 reraise(exc_type, exc_value, tb)
 File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
 func(*targs, **kargs)
 File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
 h.close()
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py", line
73, in close
 if self.closed:
AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
Error in sys.exitfunc:
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
 func(*targs, **kargs)
 File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
 h.close()
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py", line
73, in close
 if self.closed:
AttributeError: 'GCSTaskHandler' object has no attribute 'closed'

 If I look at the Airflow code I see the following code for the GcsTaskHandler:

https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/utils/log/gcs_task_handler.py

Here the closed attributed indeed refers to nowhere, does somebody know how to resolve this
issue?

 

  was:
The documentation of how to install logging to a Google Cloud bucket is unclear.

I am now following the tutorial on the airflow page:

[https://airflow.apache.org/howto/write-logs.html]

Here I find it unclear what part of the 'logger' I have to adjust in the `{{airflow/config_templates/airflow_local_settings.py}}`.

 

The adjustment states:

 
 # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of 'file.task'.
'loggers':
 Unknown macro: \{ 'airflow.task'}

 

However what I find in the template is:
|'loggers': \{\| \|'airflow.processor': { \\\\\\\\| \\\\\\\\|'handlers': ['processor'], \\\\\\\\|
\\\\\\\\|'level': LOG_LEVEL, \\\\\\\\| \\\\\\\\|'propagate': False, \\\\\\\\| \\\\\\\\|},|
|'airflow.task': { 
\| 
\|'handlers': ['task'], 
\| 
\|'level': LOG_LEVEL, 
\| 
\|'propagate': False, 
\| 
\|},|
|'flask_appbuilder': { 
\| 
\|'handler': ['console'], 
\| 
\|'level': FAB_LOG_LEVEL, 
\| 
\|'propagate': True, 
\| 
\|}|

},

 

Since for me it is very important to do it right at the first time I hope some clarity can
be provided in what has to be adjusted in the logger. Is it only the 'airflow.task' or more?

Furthermore, at step 6 it is a little unclear what remote_log_conn_id means. I would propose
to add a little more information to make this more clear.

 

The current error I am facing is:

Traceback (most recent call last):
 File "/usr/local/bin/airflow", line 16, in <module>
 from airflow import configuration
 File "/usr/local/lib/python2.7/site-packages/airflow/__init__.py", line 31, in <module>
 from airflow import settings
 File "/usr/local/lib/python2.7/site-packages/airflow/settings.py", line 198, in <module>
 configure_logging()
 File "/usr/local/lib/python2.7/site-packages/airflow/logging_config.py", line 71, in configure_logging
 dictConfig(logging_config)
 File "/usr/local/lib/python2.7/logging/config.py", line 794, in dictConfig
 dictConfigClass(config).configure()
 File "/usr/local/lib/python2.7/logging/config.py", line 568, in configure
 handler = self.configure_handler(handlers[name])
 File "/usr/local/lib/python2.7/logging/config.py", line 733, in configure_handler
 result = factory(**kwargs)
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py", line
30, in __init__
 super(GCSTaskHandler, self).__init__(base_log_folder, filename_template)
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/file_task_handler.py", line
46, in __init__
 self.filename_jinja_template = Template(self.filename_template)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 926, in __new__
 return env.from_string(source, template_class=cls)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 862, in from_string
 return cls.from_code(self, self.compile(source), globals, None)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 565, in compile
 self.handle_exception(exc_info, source_hint=source_hint)
 File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
 reraise(exc_type, exc_value, tb)
 File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
 func(*targs, **kargs)
 File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
 h.close()
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py", line
73, in close
 if self.closed:
AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
Error in sys.exitfunc:
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
 func(*targs, **kargs)
 File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
 h.close()
 File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py", line
73, in close
 if self.closed:
AttributeError: 'GCSTaskHandler' object has no attribute 'closed'

 

 

    Component/s: gcp

> Documentation for Google Logging unclear
> ----------------------------------------
>
>                 Key: AIRFLOW-3279
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3279
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: configuration, Documentation, gcp, logging
>            Reporter: Paul Velthuis
>            Priority: Blocker
>
> The documentation of how to install logging to a Google Cloud bucket is unclear.
> I am now following the tutorial on the airflow page:
> [https://airflow.apache.org/howto/write-logs.html]
> Here I find it unclear what part of the 'logger' I have to adjust in the `{{airflow/config_templates/airflow_local_settings.py}}`.
>  
> The adjustment states:
>  
>  # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of
'file.task'. 'loggers':
>  Unknown macro: \{ 'airflow.task'}
>  
> However what I find in the template is:
> |'loggers': \{\| \|'airflow.processor': { \\\\\\\\| \\\\\\\\|'handlers': ['processor'],
\\\\\\\\| \\\\\\\\|'level': LOG_LEVEL, \\\\\\\\| \\\\\\\\|'propagate': False, \\\\\\\\| \\\\\\\\|},|
> |'airflow.task': { 
> \| 
> \|'handlers': ['task'], 
> \| 
> \|'level': LOG_LEVEL, 
> \| 
> \|'propagate': False, 
> \| 
> \|},|
> |'flask_appbuilder': { 
> \| 
> \|'handler': ['console'], 
> \| 
> \|'level': FAB_LOG_LEVEL, 
> \| 
> \|'propagate': True, 
> \| 
> \|}|
> },
>  
> Since for me it is very important to do it right at the first time I hope some clarity
can be provided in what has to be adjusted in the logger. Is it only the 'airflow.task' or
more?
> Furthermore, at step 6 it is a little unclear what remote_log_conn_id means. I would
propose to add a little more information to make this more clear.
>  
> The current error I am facing is:
> Traceback (most recent call last):
>  File "/usr/local/bin/airflow", line 16, in <module>
>  from airflow import configuration
>  File "/usr/local/lib/python2.7/site-packages/airflow/__init__.py", line 31, in <module>
>  from airflow import settings
>  File "/usr/local/lib/python2.7/site-packages/airflow/settings.py", line 198, in <module>
>  configure_logging()
>  File "/usr/local/lib/python2.7/site-packages/airflow/logging_config.py", line 71, in
configure_logging
>  dictConfig(logging_config)
>  File "/usr/local/lib/python2.7/logging/config.py", line 794, in dictConfig
>  dictConfigClass(config).configure()
>  File "/usr/local/lib/python2.7/logging/config.py", line 568, in configure
>  handler = self.configure_handler(handlers[name])
>  File "/usr/local/lib/python2.7/logging/config.py", line 733, in configure_handler
>  result = factory(**kwargs)
>  File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
line 30, in __init__
>  super(GCSTaskHandler, self).__init__(base_log_folder, filename_template)
>  File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/file_task_handler.py",
line 46, in __init__
>  self.filename_jinja_template = Template(self.filename_template)
>  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 926, in __new__
>  return env.from_string(source, template_class=cls)
>  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 862, in from_string
>  return cls.from_code(self, self.compile(source), globals, None)
>  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 565, in compile
>  self.handle_exception(exc_info, source_hint=source_hint)
>  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
>  reraise(exc_type, exc_value, tb)
>  File "<unknown>", line 1, in template
> jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
> Error in atexit._run_exitfuncs:
> Traceback (most recent call last):
>  File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
>  func(*targs, **kargs)
>  File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
>  h.close()
>  File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
line 73, in close
>  if self.closed:
> AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
> Error in sys.exitfunc:
> Traceback (most recent call last):
>  File "/usr/local/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
>  func(*targs, **kargs)
>  File "/usr/local/lib/python2.7/logging/__init__.py", line 1676, in shutdown
>  h.close()
>  File "/usr/local/lib/python2.7/site-packages/airflow/utils/log/gcs_task_handler.py",
line 73, in close
>  if self.closed:
> AttributeError: 'GCSTaskHandler' object has no attribute 'closed'
>  If I look at the Airflow code I see the following code for the GcsTaskHandler:
> https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/utils/log/gcs_task_handler.py
> Here the closed attributed indeed refers to nowhere, does somebody know how to resolve
this issue?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message