airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2363) S3 remote logging appending tuple instead of str
Date Thu, 26 Apr 2018 21:48:07 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454925#comment-16454925
] 

Kevin Yang commented on AIRFLOW-2363:
-------------------------------------

[~b11c] If the case is that the set_context is not called properly then this will not resolve
2379.

 

The entry point of the task handler's set_context() method should be here: [https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py#L460,] which
is called in all `airflow run` command. Not sure why it can be missed.

 

I kinda suspect the reason log not being uploaded is because of the bug getting fixed in this
issue but I might need your help to confirm that.

> S3 remote logging appending tuple instead of str
> ------------------------------------------------
>
>                 Key: AIRFLOW-2363
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2363
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: logging
>            Reporter: Kyle Hamlin
>            Priority: Major
>             Fix For: 1.10.0
>
>
> A recent merge into master that added support for Elasticsearch logging seems to have
broken S3 logging by returning a tuple instead of a string.
> [https://github.com/apache/incubator-airflow/commit/ec38ba9594395de04ec932481212a86fbe9ae107#diff-0442332ecbe42ebbf426911c68d8cd4aR128]
>  
> following errors thrown:
>  
> *Session NoneType error*
>  Traceback (most recent call last):
>    File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/s3_task_handler.py",
line 171, in s3_write
>      encrypt=configuration.conf.getboolean('core', 'ENCRYPT_S3_LOGS'),
>    File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 274,
in load_string
>      encrypt=encrypt)
>    File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 313,
in load_bytes
>      client = self.get_conn()
>    File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 34,
in get_conn
>      return self.get_client_type('s3')
>    File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py",
line 151, in get_client_type
>      session, endpoint_url = self._get_credentials(region_name)
>    File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py",
line 97, in _get_credentials
>      connection_object = self.get_connection(self.aws_conn_id)
>    File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 82,
in get_connection
>      conn = random.choice(cls.get_connections(conn_id))
>    File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 77,
in get_connections
>      conns = cls._get_connections_from_db(conn_id)
>    File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 72, in wrapper
>      with create_session() as session:
>    File "/usr/local/lib/python3.6/contextlib.py", line 81, in __enter__
>      return next(self.gen)
>    File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 41, in create_session
>      session = settings.Session()
>  TypeError: 'NoneType' object is not callable
>  
> *TypeError must be str not tuple*
>  [2018-04-16 18:37:28,200] ERROR in app: Exception on /admin/airflow/get_logs_with_metadata
[GET]
>  Traceback (most recent call last):
>    File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
>      response = self.full_dispatch_request()
>    File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
>      rv = self.handle_user_exception(e)
>    File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
>      reraise(exc_type, exc_value, tb)
>    File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
>      raise value
>    File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
>      rv = self.dispatch_request()
>    File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
>      return self.view_functions[rule.endpoint](**req.view_args)
>    File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
>      return self._run_view(f, *args, **kwargs)
>    File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368, in
_run_view
>      return fn(self, *args, **kwargs)
>    File "/usr/local/lib/python3.6/site-packages/flask_login.py", line 755, in decorated_view
>      return func(*args, **kwargs)
>    File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 269, in
wrapper
>      return f(*args, **kwargs)
>    File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
>      return func(*args, **kwargs)
>    File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 770, in
get_logs_with_metadata
>      logs, metadatas = handler.read(ti, try_number, metadata=metadata)
>    File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_task_handler.py",
line 165, in read
>      logs[i] += log
>  TypeError: must be str, not tuple



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message