airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ace Haidrey (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1735) Log files do not show up for unscheduled dags
Date Thu, 19 Oct 2017 03:26:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ace Haidrey updated AIRFLOW-1735:
---------------------------------
    Description: 
I created a new dag with a schedule of {{@once}} and left it unscheduled.  I ran different
tasks a few times from the UI but the logs never showed up.  Instead I'd see messages like
the following:
{code}
*** Log file isn't local.
*** Fetching here: http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}

There is a log file for this task, but it seems to be at a slightly different path on the
machine:

{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}

And indeed, when I change the URL to WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request&dag_id=DAG_ID&execution_date=2017-06-21T00:00:00,
I see the actual log:

{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from ~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava app IDs 
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}

I've added a screen shot of it below.

*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00 and we can
get that quite easily if it doesnt have it. You can see in this PR I made, but basically if
we set
{code}
dttm = dateutil.parser.parse(request.args.get('execution_date'))
execution_date = dttm.isoformat()
{code}
This will fix it.

  was:
I created a new dag with a schedule of {{@once}} and left it unscheduled.  I ran different
tasks a few times from the UI but the logs never showed up.  Instead I'd see messages like
the following:
{code}
*** Log file isn't local.
*** Fetching here: http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}

There is a log file for this task, but it seems to be at a slightly different path on the
machine:

{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}

And indeed, when I change the URL to WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request&dag_id=DAG_ID&execution_date=2017-06-21T00:00:00,
I see the actual log:

{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from ~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava app IDs 
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}

I've added a screen shot of it below.

*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00 and we can
get that quite easily if it doesnt have it. You can see in this PR I made.


> Log files do not show up for unscheduled dags
> ---------------------------------------------
>
>                 Key: AIRFLOW-1735
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Ace Haidrey
>            Assignee: Ace Haidrey
>              Labels: www
>         Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png
>
>
> I created a new dag with a schedule of {{@once}} and left it unscheduled.  I ran different
tasks a few times from the UI but the logs never showed up.  Instead I'd see messages like
the following:
> {code}
> *** Log file isn't local.
> *** Fetching here: http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
> *** Failed to fetch log file from worker.
> {code}
> There is a log file for this task, but it seems to be at a slightly different path on
the machine:
> {code}
> (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
> -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
> {code}
> And indeed, when I change the URL to WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request&dag_id=DAG_ID&execution_date=2017-06-21T00:00:00,
I see the actual log:
> {code}
> *** Reading local log.
> [2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from ~/analytics_kochava_import_dag.py
> [2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava app IDs

> ...
> [2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
> {code}
> I've added a screen shot of it below.
> *FIX:*
> I have noticed in www/view.py the issue is not have the time component 00:00:00 and we
can get that quite easily if it doesnt have it. You can see in this PR I made, but basically
if we set
> {code}
> dttm = dateutil.parser.parse(request.args.get('execution_date'))
> execution_date = dttm.isoformat()
> {code}
> This will fix it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message