From commits-return-14316-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Thu Apr 26 23:40:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id CF518180679 for ; Thu, 26 Apr 2018 23:40:05 +0200 (CEST) Received: (qmail 91168 invoked by uid 500); 26 Apr 2018 21:40:04 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 91107 invoked by uid 99); 26 Apr 2018 21:40:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Apr 2018 21:40:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6001AC0254 for ; Thu, 26 Apr 2018 21:40:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id y3y4qNgS9UV6 for ; Thu, 26 Apr 2018 21:40:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 34C6D5F5B2 for ; Thu, 26 Apr 2018 21:40:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 210B3E1290 for ; Thu, 26 Apr 2018 21:40:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 756B121094 for ; Thu, 26 Apr 2018 21:40:00 +0000 (UTC) Date: Thu, 26 Apr 2018 21:40:00 +0000 (UTC) From: "Berislav Lopac (JIRA)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (AIRFLOW-2363) S3 remote logging appending tuple instead of str MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-2363?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 454914#comment-16454914 ]=20 Berislav Lopac edited comment on AIRFLOW-2363 at 4/26/18 9:39 PM: ------------------------------------------------------------------ Is there any chance that this might also resolve AIRFLOW-2379? was (Author: b11c): Is there any chance that this might also resolve AIRFLOW-2375? > S3 remote logging appending tuple instead of str > ------------------------------------------------ > > Key: AIRFLOW-2363 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2363 > Project: Apache Airflow > Issue Type: Bug > Components: logging > Reporter: Kyle Hamlin > Priority: Major > Fix For: 1.10.0 > > > A recent merge into master that added support for=C2=A0Elasticsearch logg= ing seems to have broken S3 logging by returning a tuple instead of a strin= g. > [https://github.com/apache/incubator-airflow/commit/ec38ba9594395de04ec93= 2481212a86fbe9ae107#diff-0442332ecbe42ebbf426911c68d8cd4aR128] > =C2=A0 > following errors thrown: > =C2=A0 > *Session NoneType error* > =C2=A0Traceback (most recent call last): > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/utils/l= og/s3_task_handler.py", line 171, in s3_write > =C2=A0 =C2=A0 =C2=A0encrypt=3Dconfiguration.conf.getboolean('core', 'ENCR= YPT_S3_LOGS'), > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S= 3_hook.py", line 274, in load_string > =C2=A0 =C2=A0 =C2=A0encrypt=3Dencrypt) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S= 3_hook.py", line 313, in load_bytes > =C2=A0 =C2=A0 =C2=A0client =3D self.get_conn() > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S= 3_hook.py", line 34, in get_conn > =C2=A0 =C2=A0 =C2=A0return self.get_client_type('s3') > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/contrib= /hooks/aws_hook.py", line 151, in get_client_type > =C2=A0 =C2=A0 =C2=A0session, endpoint_url =3D self._get_credentials(regio= n_name) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/contrib= /hooks/aws_hook.py", line 97, in _get_credentials > =C2=A0 =C2=A0 =C2=A0connection_object =3D self.get_connection(self.aws_co= nn_id) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/hooks/b= ase_hook.py", line 82, in get_connection > =C2=A0 =C2=A0 =C2=A0conn =3D random.choice(cls.get_connections(conn_id)) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/hooks/b= ase_hook.py", line 77, in get_connections > =C2=A0 =C2=A0 =C2=A0conns =3D cls._get_connections_from_db(conn_id) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/utils/d= b.py", line 72, in wrapper > =C2=A0 =C2=A0 =C2=A0with create_session() as session: > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/contextlib.py", line 81, in _= _enter__ > =C2=A0 =C2=A0 =C2=A0return next(self.gen) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/utils/d= b.py", line 41, in create_session > =C2=A0 =C2=A0 =C2=A0session =3D settings.Session() > =C2=A0TypeError: 'NoneType' object is not callable > =C2=A0 > *TypeError must be str not tuple* > =C2=A0[2018-04-16 18:37:28,200] ERROR in app: Exception on /admin/airflow= /get_logs_with_metadata [GET] > =C2=A0Traceback (most recent call last): > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask/app.py", = line 1982, in wsgi_app > =C2=A0 =C2=A0 =C2=A0response =3D self.full_dispatch_request() > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask/app.py", = line 1614, in full_dispatch_request > =C2=A0 =C2=A0 =C2=A0rv =3D self.handle_user_exception(e) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask/app.py", = line 1517, in handle_user_exception > =C2=A0 =C2=A0 =C2=A0reraise(exc_type, exc_value, tb) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask/_compat.p= y", line 33, in reraise > =C2=A0 =C2=A0 =C2=A0raise value > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask/app.py", = line 1612, in full_dispatch_request > =C2=A0 =C2=A0 =C2=A0rv =3D self.dispatch_request() > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask/app.py", = line 1598, in dispatch_request > =C2=A0 =C2=A0 =C2=A0return self.view_functions[rule.endpoint](**req.view_= args) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask_admin/bas= e.py", line 69, in inner > =C2=A0 =C2=A0 =C2=A0return self._run_view(f, *args, **kwargs) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask_admin/bas= e.py", line 368, in _run_view > =C2=A0 =C2=A0 =C2=A0return fn(self, *args, **kwargs) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/flask_login.py"= , line 755, in decorated_view > =C2=A0 =C2=A0 =C2=A0return func(*args, **kwargs) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/www/uti= ls.py", line 269, in wrapper > =C2=A0 =C2=A0 =C2=A0return f(*args, **kwargs) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/utils/d= b.py", line 74, in wrapper > =C2=A0 =C2=A0 =C2=A0return func(*args, **kwargs) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/www/vie= ws.py", line 770, in get_logs_with_metadata > =C2=A0 =C2=A0 =C2=A0logs, metadatas =3D handler.read(ti, try_number, meta= data=3Dmetadata) > =C2=A0 =C2=A0File "/usr/local/lib/python3.6/site-packages/airflow/utils/l= og/file_task_handler.py", line 165, in read > =C2=A0 =C2=A0 =C2=A0logs[i] +=3D log > =C2=A0TypeError: must be str, not tuple -- This message was sent by Atlassian JIRA (v7.6.3#76005)