From users-return-423-archive-asf-public=cust-asf.ponee.io@airflow.apache.org  Thu Sep  3 09:52:24 2020
Return-Path: <users-return-423-archive-asf-public=cust-asf.ponee.io@airflow.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153])
	by mx-eu-01.ponee.io (Postfix) with ESMTPS id BF3FF180636
	for <archive-asf-public@cust-asf.ponee.io>; Thu,  3 Sep 2020 11:52:24 +0200 (CEST)
Received: from mail.apache.org (localhost [127.0.0.1])
	by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id E380D121B43
	for <archive-asf-public@cust-asf.ponee.io>; Thu,  3 Sep 2020 09:52:23 +0000 (UTC)
Received: (qmail 84037 invoked by uid 500); 3 Sep 2020 09:52:23 -0000
Mailing-List: contact users-help@airflow.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@airflow.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@airflow.apache.org>
List-Post: <mailto:users@airflow.apache.org>
List-Id: <users.airflow.apache.org>
Reply-To: users@airflow.apache.org
Delivered-To: mailing list users@airflow.apache.org
Received: (qmail 84018 invoked by uid 99); 3 Sep 2020 09:52:23 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2020 09:52:23 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 7CC73C0C13;
	Thu,  3 Sep 2020 09:52:22 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -2.299
X-Spam-Level:
X-Spam-Status: No, score=-2.299 tagged_above=-999 required=6.31
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_MED=-2.3,
	RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
	SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001]
	autolearn=disabled
Authentication-Results: spamd1-us-west.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=sfr.com header.b=csugsnqU;
	dkim=pass (1024-bit key) header.d=globaltelko.onmicrosoft.com
	header.b=WBuv3qXv
Received: from mx1-ec2-va.apache.org ([10.40.0.8])
	by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024)
	with ESMTP id nRIXjR_e8jFQ; Thu,  3 Sep 2020 09:52:18 +0000 (UTC)
Received-SPF: Pass (helo) identity=helo; client-ip=77.154.156.201; helo=mx3-corp.sfr.com; envelope-from=prvs=508acd8a7=kevin.joaquim@sfr.com; receiver=<UNKNOWN> 
Received: from mx3-corp.sfr.com (mx3-corp.sfr.com [77.154.156.201])
	by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id CBD74BC27D;
	Thu,  3 Sep 2020 09:52:14 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=sfr.com; q=dns/txt; s=s2048; t=1599126738;
  x=1599731538;
  h=from:to:subject:date:message-id:mime-version;
  bh=WbJ2FCc01Eb2UnOYihq+HROJE1T+OARbdVw0yp09WXg=;
  b=csugsnqUkVPfRW+krwk/tuKoEYffBm2wOdsBz8H8fS1VTlKj/EnNdxDy
   cRzPQXPPgCzqdQVctjpadzcgDzZBF/0Ei5W3+W5rt8GZzQjuhgFo0JjEU
   blrN0nz1fxezHT9Ev309lRsYyp8M8uNiG1IaccwQtzk+yt5dI+eSbu0SL
   LEoHYn9rkGfXmCcX9l0O/bEYy3p1NFPsZcadwzIX7D7nnvTmUGv3a8lwl
   Qok8/9QZNW/D3ke/QqcXbtQiCCCt4UcYixLGkE6eCns3K5wFLEhe4CNB4
   yUJ5efIRZAQaD7KB+2vIYzjVXE8ElYgOFzAQQCTJOaxE+6ypY5KhOyKOc
   Q==;
X-IronPort-AV: E=Sophos;i="5.76,359,1592863200"; 
   d="scan'208,217";a="114630595"
Received: from unknown (HELO EXCH012.encara.local.ads) ([10.29.105.1])
  by mx3int-corp.prod with ESMTP; 03 Sep 2020 11:52:06 +0200
Received: from EXCH013.encara.local.ads (10.29.105.107) by
 EXCH012.encara.local.ads (10.29.105.1) with Microsoft SMTP Server (TLS) id
 14.3.487.0; Thu, 3 Sep 2020 11:52:06 +0200
Received: from EXCE051.encara.local.ads (10.163.136.161) by
 EXCH013.encara.local.ads (10.29.105.107) with Microsoft SMTP Server (TLS) id
 14.3.487.0; Thu, 3 Sep 2020 11:52:06 +0200
Received: from FRA01-PR2-obe.outbound.protection.outlook.com (104.47.24.104)
 by exchedge04.sfr.com (80.125.162.161) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.1.1713.5; Thu, 3 Sep 2020 11:52:06 +0200
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=A9viOhII5NnMUGnfIH59eNz+XUj+9DvcHSvN0woeezEFkC7Hb1S4ghAT5qOvok7I5nSs54ODR18rj2Nexurye7onMVEviltRNqCX4vHIW3P/VNoctNLU3OZ4g4cVbrlY+8+YBRKz7malhm6eNSX6+stqhthoyLMosq1oNhQd6fcaGrdLqnLTUpPDgTb8ObgkKFBzQ5IZ5LHIOi9+4hsgRkwqauOvfuG8A1R/Hkk7Op2HXeJGv0FJk+hU2KPy6BC+UYVf+lDpYi3i8TChJhlLgskOtUpiyXW2vX6hoqDlDUd81pn4c7sqTsnsKMY7sg6xHT7RvJGgYuX5VeTqWYX/yg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=WbJ2FCc01Eb2UnOYihq+HROJE1T+OARbdVw0yp09WXg=;
 b=YIKZ+SWcf35n9unkcAG5cQmh3/iQXkw5cwEgYXuvKicEOgNNSpYSlb/eIuTRRhq3vS/nxQxORKwjq7nAUEHkSdAxcbWfvuusUrzfZZniGj+IB7GdZIo0hS9ZCZgS7mprjIPofnk78o4Z8/p1qFuPsAbo49AQdbMlNIyCX0hef1VI2+JbER+sHHw/ko5QpzvajCsjW8xywMhxUL78QbhHHq5poQE8w8L361b+JlzVVxJktdDkN7QntmGO+FQyagnexE7PlnrbitXN47DgMjCNpr59BFhh5Dktp7W/yFIwpqnNCm1R3Ygfr0B5C5AJ4Ah8raivy4F1egwlgaheXE1iUQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=sfr.com; dmarc=pass action=none header.from=sfr.com; dkim=pass
 header.d=sfr.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=globaltelko.onmicrosoft.com; s=selector1-globaltelko-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=WbJ2FCc01Eb2UnOYihq+HROJE1T+OARbdVw0yp09WXg=;
 b=WBuv3qXvWsoKs86Za+UE4ZLMYK2lDWWUjy9Uiz4gh95wUddmNtCdTeBnXbnfpRwYcb52B+SxLohLLT53BGGYoqP5JA22K3Kl6GJXZZZUA3jfT1d4SGJJ3NO8Fggwy67q/eLZNXHVyLav609dfa+D/oskaAHFjHPyhmwpEHXc93c=
Received: from MR2P264MB0468.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:6::20) by
 MRXP264MB0167.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:20::21) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3348.15; Thu, 3 Sep 2020 09:51:51 +0000
Received: from MR2P264MB0468.FRAP264.PROD.OUTLOOK.COM
 ([fe80::e087:5b3b:ca7a:9f63]) by MR2P264MB0468.FRAP264.PROD.OUTLOOK.COM
 ([fe80::e087:5b3b:ca7a:9f63%6]) with mapi id 15.20.3326.025; Thu, 3 Sep 2020
 09:51:51 +0000
From: "JOAQUIM, Kevin" <kevin.joaquim@sfr.com>
To: "users@airflow.apache.org" <users@airflow.apache.org>,
	"dev@airflow.apache.org" <dev@airflow.apache.org>, !Admin-ordo
	<adminordo@sfr.com>, "LEVESQUE, Eric" <eric.levesque@sfr.com>
Subject: KubernetesPodOperator ERROR - Connection broken but pod write log
Thread-Topic: KubernetesPodOperator ERROR - Connection broken but pod write
 log
Thread-Index: AQHWgdEjPfhVPIfBakuu7LEWaOEbOQ==
Date: Thu, 3 Sep 2020 09:51:51 +0000
Message-ID: <MR2P264MB04687A677C05CCB6F5700A22E72C0@MR2P264MB0468.FRAP264.PROD.OUTLOOK.COM>
Accept-Language: fr-FR, en-US
Content-Language: fr-FR
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: airflow.apache.org; dkim=none (message not signed)
 header.d=none;airflow.apache.org; dmarc=none action=none header.from=sfr.com;
x-originating-ip: [217.70.85.225]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: b630a107-fa19-4324-3138-08d84feefba5
x-ms-traffictypediagnostic: MRXP264MB0167:
x-ld-processed: d92fdf0e-a2b6-4bf4-a810-0f4961cdb50f,ExtAddr
x-ms-exchange-transport-forked: True
x-microsoft-antispam-prvs: <MRXP264MB016730A4EDC248D6F14B2349E72C0@MRXP264MB0167.FRAP264.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:8273;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: eJ23ZAi4PZ7H+IupKCS2v/yH0LCuHMiKtGpEBlGat4eeykVqwOUiCsGKTK8MOVm0bdIxmXIrGhmfZi1J/ziXllP42ZzRNmM8hFuuvmiiBiqBLHFXOp7jIF04GHh0QOKfVosYkQ/7xtIMU1lsxMpVrj6Yj4EQ4bToPbcecedU6cdjijfhs+fja8Wlj4UMJ+/EYyq/JCrLG8KE4VWOPE6a0NtbrDSOMsfWaS61lxNohSUgMinkNnfS47r26dyWZPP1HQ3WNXoB3ZUHYpbBUfEDa2K5yUJF8CMjFvpBZAOpdwEU/bIe0HUtQYhVSIRTj9bwXn6cJ05ciuw52V5KcTL+QNy/R6qOXzRE9sUCqxDE6eEj6EUK6d7LCOaTqSQdhwcqX0ZWsPXeM6pEnNlkK232cQ==
x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MR2P264MB0468.FRAP264.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(4636009)(136003)(366004)(396003)(346002)(39860400002)(376002)(166002)(33656002)(71200400001)(4744005)(9686003)(86362001)(26005)(186003)(6636002)(6506007)(7696005)(450100002)(316002)(5660300002)(83380400001)(76116006)(91956017)(8936002)(478600001)(66446008)(110136005)(66476007)(55016002)(19627405001)(52536014)(8676002)(2906002)(64756008)(66556008)(966005)(66946007);DIR:OUT;SFP:1101;
x-ms-exchange-antispam-messagedata: SorW5X3ZOF+LGozHfo9iKKkZUFZZNDeWZxlMZZkVWNf0UbmzEiCh3VqRe/+8ZboRB/4QGwaPKujekg13eBMFFzQcO0hqnVd98k+lcYeoFqkjHer8QAagoAP5DCdsf/tiNSvi+y1V0cvcSRo6QGNKeh6/0bUPi7IdhWjb3/LJkx2G5jNjexZGka7UE+M9wGEVgDL34nE2tka3f0VXSXbRAKcavqQgp9e08O6BkYMn0X9NzWX2ohVx7nd5cNBE73Df2/FacVhM15l8ZnNvmMxZWj84uGVAZl+AjqFYcmIxrrRbcm1PglGJfHpuwOWW1jqbh/JLc3emHFc1mkCSeoE4PEXo7Qd1PfXIeI0CAUB90aUv+/MLdWtZhMNz4Nr382f+vbN6SdbPAYTF+5N6EY2/tsACYPRi93900G0743/7feQxruRZQSRhnNHNSM/lH8w+6F2Xs5o9UA3b68W/YoN4T3Hp9TV1Ed1jn7rPBvpwRvL68s3EH5EJQtEQLIRtwAJSrmRy6Cs+Atp/Sa+z6BOpE2pe4egZs32wSVSTUEB8ys2wsYmK+dMaTqF9BOtUFug9jPuin3szign05GOn4hzHb30tEU/998YtJMMgyUZjnhyM5Msa8qNK/v0ZS0Rk9OJf1cd62Rtw/Ndp2VWB8mERLg==
Content-Type: multipart/alternative;
	boundary="_000_MR2P264MB04687A677C05CCB6F5700A22E72C0MR2P264MB0468FRAP_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MR2P264MB0468.FRAP264.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-Network-Message-Id: b630a107-fa19-4324-3138-08d84feefba5
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Sep 2020 09:51:51.4493
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: d92fdf0e-a2b6-4bf4-a810-0f4961cdb50f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: uLmKY8ZgTbT6YC1UH/IiKobpYVGNArTNZTPWwcD38ZDNgdgZfSpeHvg6b64JvaQjCRju54u4RAipKGvjuE9KkQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MRXP264MB0167
X-OriginatorOrg: sfr.com

--_000_MR2P264MB04687A677C05CCB6F5700A22E72C0MR2P264MB0468FRAP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi team,

I have a ERROR in production on my KubernetesPodOperator tasks with log : E=
RROR - Connection broken.

when my job is running, my pod writes logs visible in the airflow interface=
. But i don't know why airflow no longer retrieves the logs from my pod and=
 passes the task into zombie. After the time of scheduler_zombie_task_thres=
hold the job is killed. But when I log into my pod, i can see it is writing=
 logs and there is activity. the network and the cluster k8s is ok..

I found this issue: https://issues.apache.org/jira/browse/AIRFLOW-3534 , bu=
t my pod writes continuous logs on kubernetes (stdout)

Why Airflow stops collecting my task log KubernetesPodOperator ? And why ai=
rflow monitor the status of the task with the log collection ..?

I can set get_logs to False, but that doesn't solve my problem.. I'm using =
Airflow v1.10.10.

Who can help me? Thanks in advance.

Regards,

Kevin JOAQUIM

--_000_MR2P264MB04687A677C05CCB6F5700A22E72C0MR2P264MB0468FRAP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
Hi team,</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
I have a ERROR in production on my KubernetesPodOperator tasks with log :&n=
bsp;ERROR - Connection broken.<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
when my job is running, my pod writes logs visible in the airflow interface=
. But i don't know why airflow no longer retrieves the logs from my pod and=
 passes the task into zombie. After the time of scheduler_zombie_task_thres=
hold the job is killed. B<span style=3D"color: rgb(0, 0, 0); font-family: C=
alibri, Arial, Helvetica, sans-serif; font-size: 12pt;">ut
 when I log into my pod, i can see it is writing logs and there is activity=
.&nbsp;the network and the cluster k8s is ok..</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica,=
 sans-serif; font-size: 12pt;"><br>
</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica,=
 sans-serif; font-size: 12pt;">I found this issue:&nbsp;<a href=3D"https://=
issues.apache.org/jira/browse/AIRFLOW-3534" id=3D"LPlnk525657">https://issu=
es.apache.org/jira/browse/AIRFLOW-3534</a>
 , but my pod writes continuous logs on&nbsp;kubernetes (stdout)<br>
</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
Why Airflow stops collecting my task log&nbsp;<span style=3D"font-family: C=
alibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255);=
 display: inline !important">KubernetesPodOperator ? And&nbsp;why airflow m=
onitor the status of the task with the log collection
 ..?</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; backgrou=
nd-color: rgb(255, 255, 255); display: inline !important">I can set get_log=
s to False,&nbsp;but that doesn't solve my problem.. I'm using Airflow v1.1=
0.10.<br>
</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; backgrou=
nd-color: rgb(255, 255, 255); display: inline !important"><br>
</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; backgrou=
nd-color: rgb(255, 255, 255); display: inline !important">Who can help me?
<span style=3D"font-family: Calibri, sans-serif; background-color: rgb(255,=
 255, 255); display: inline !important">
Thanks in advance.</span><br>
</span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; backgrou=
nd-color: rgb(255, 255, 255); display: inline !important"><span style=3D"fo=
nt-family: Calibri, sans-serif; background-color: rgb(255, 255, 255); displ=
ay: inline !important"><br>
</span></span></div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);">
<span style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; backgrou=
nd-color: rgb(255, 255, 255); display: inline !important"><span style=3D"fo=
nt-family: Calibri, sans-serif; background-color: rgb(255, 255, 255); displ=
ay: inline !important">Regards,
<div><br>
</div>
Kevin JOAQUIM <br>
</span></span></div>
</body>
</html>

--_000_MR2P264MB04687A677C05CCB6F5700A22E72C0MR2P264MB0468FRAP_--