airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1873) Task operator logs appear in wrong numbered log file
Date Thu, 07 Dec 2017 13:42:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281858#comment-16281858
] 

ASF subversion and git services commented on AIRFLOW-1873:
----------------------------------------------------------

Commit 9a552c4fe88d362f79d644ec04340b7b63af795f in incubator-airflow's branch refs/heads/v1-9-test
from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9a552c4 ]

[AIRFLOW-1873] Set TI.try_number to right value depending TI state

Rather than having try_number+1 in various places,
try_number
will now automatically contain the right value for
when the TI
will next be run, and handle the case where
try_number is
accessed when the task is currently running.

This showed up as a bug where the logs from
running operators would
show up in the next log file (2.log for the first
try)

Closes #2832 from ashb/AIRFLOW-1873-task-operator-
log-try-number

(cherry picked from commit 4b4e504eeae81e48f3c9d796a61dd9e86000c663)
Signed-off-by: Bolke de Bruin <bolke@xs4all.nl>


> Task operator logs appear in wrong numbered log file
> ----------------------------------------------------
>
>                 Key: AIRFLOW-1873
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1873
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.9.0
>            Reporter: Ash Berlin-Taylor
>             Fix For: 1.9.0
>
>
> The logs for the running operators appear in the "next" task number.
> For example, for the first try for a given task instance the "collecting dag" etc appear
in 1.log, but log messages from the operator itself appear in 2.log.
> 1.log:
> {noformat}
> [2017-11-30 23:14:44,189] {cli.py:374} INFO - Running on host 4f1698e8ae61
> [2017-11-30 23:14:44,254] {models.py:1173} INFO - Dependencies all met for <TaskInstance:
tests.test-logging 2017-11-20 00:00:00 [queued]>
> [2017-11-30 23:14:44,265] {models.py:1173} INFO - Dependencies all met for <TaskInstance:
tests.test-logging 2017-11-20 00:00:00 [queued]>
> [2017-11-30 23:14:44,266] {models.py:1383} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2017-11-30 23:14:44,290] {models.py:1404} INFO - Executing <Task(PythonOperator):
test-logging> on 2017-11-20 00:00:00
> [2017-11-30 23:14:44,291] {base_task_runner.py:115} INFO - Running: ['bash', '-c', 'airflow
run tests test-logging 2017-11-20T00:00:00 --job_id 4 --raw -sd /usr/local/airflow/dags/example/csv_to_parquet.py']
> [2017-11-30 23:14:50,054] {base_task_runner.py:98} INFO - Subtask: [2017-11-30 23:14:50,052]
{configuration.py:206} WARNING - section/key [celery/celery_ssl_active] not found in config
> [2017-11-30 23:14:50,056] {base_task_runner.py:98} INFO - Subtask: [2017-11-30 23:14:50,052]
{default_celery.py:41} WARNING - Celery Executor will run without SSL
> [2017-11-30 23:14:50,058] {base_task_runner.py:98} INFO - Subtask: [2017-11-30 23:14:50,054]
{__init__.py:45} INFO - Using executor CeleryExecutor
> [2017-11-30 23:14:50,529] {base_task_runner.py:98} INFO - Subtask: [2017-11-30 23:14:50,529]
{models.py:189} INFO - Filling up the DagBag from /usr/local/airflow/dags/example/csv_to_parquet.py
> [2017-11-30 23:14:50,830] {base_task_runner.py:98} INFO - Subtask: [2017-11-30 23:14:50,825]
{python_operator.py:90} INFO - Done. Returned value was: None
> {noformat}
> 2.log:
> {noformat}
> [2017-11-30 23:14:50,749] {cli.py:374} INFO - Running on host 4f1698e8ae61
> [2017-11-30 23:14:50,820] {logging_mixin.py:84} INFO - Hi from /usr/local/airflow/dags/example/csv_to_parquet.py
> [2017-11-30 23:14:50,824] {csv_to_parquet.py:21} ERROR - Hello
> {noformat}
> Notice the timestamps - the contents of 2.log appear just before the last line of 1.log,
and should be in the same log file (there is only a single run of this task instance)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message