airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Meickle (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-3305) KubernetesPodOperator has a race condition for log output
Date Tue, 06 Nov 2018 16:23:00 GMT
James Meickle created AIRFLOW-3305:
--------------------------------------

             Summary: KubernetesPodOperator has a race condition for log output
                 Key: AIRFLOW-3305
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3305
             Project: Apache Airflow
          Issue Type: Bug
          Components: kubernetes
    Affects Versions: 1.10.0
            Reporter: James Meickle


The KubernetesPodOperator follows logs from the container in the pod that it launches: [https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py#L96]

This is set to "follow" mode, which streams logs. However, it is possible (but not guaranteed)
for the pod's container to have started before the log stream call reaches the cluster. In
this case, re-running the same task may result in very different-looking logs, with no notification
that there was any truncation. This is a confusing experience for operators who are not familiar
with Kubernetes.

My recommendation is to remove "tail_lines" which should have the effect of fetching all
previous logs when streaming starts: https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#read_namespaced_pod_log



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message