airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1111) Make logs easier to read and separate infra/task logs
Date Fri, 14 Apr 2017 18:00:43 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dan Davydov updated AIRFLOW-1111:
---------------------------------
    Description: 
Ideally there would be tabs for logs:
Dag-parsing (needs a better name)
1 (Short for "Run 1")
2
...

Dag-parsing would contain the output like:
{code}
[2017-04-14 15:24:54,341] {models.py:168} INFO - Filling up the DagBag from /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,342] {models.py:253} DEBUG - Importing /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_r>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_large>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_tiny>
[2017-04-14 15:24:54,357] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_platinum_medium>
[2017-04-14 15:24:54,553] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup memory in
/ since it already exists
[2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup airflow in
/memory since it already exists
[2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup 2017-04-14
in /memory/airflow since it already exists
[2017-04-14 15:24:54,554] {cgroup_task_runner.py:77} DEBUG - Creating cgroup 82d42ce2-2126-11e7-ad73-22000a896f7d
in /memory/airflow/2017-04-14
{code}

and "1" would contain
{code}
2017-04-14 15:24:56,146] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,146]
{models.py:1344} INFO - Executing <Task(HiveOperator): insert_canary_row> on 2017-04-14
15:10:00
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,172]
{hive_operator.py:98} INFO - Executing: USE datainfra;
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask:           CREATE TABLE
IF NOT EXISTS datainfra.airflow_precious_canary_gold_large (key STRING);
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: INSERT OVERWRITE TABLE
datainfra.airflow_precious_canary_gold_large
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: SELECT key FROM datainfra.one_row
LIMIT 1;
[2017-04-14 15:24:56,187] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,186]
{hive_hooks.py:207} INFO - hive -hiveconf airflow.ctx.dag.dag_id=airflow_precious_canary_gold_large
-hiveconf airflow.ctx.dag_run.execution_date=2017-04-14T15:10:00 -hiveconf airflow.ctx.task_instance.execution_date=2017-04-14T15:10:00
-hiveconf airflow.ctx.task.task_id=insert_canary_row -hiveconf mapreduce.job.queuename=airflow_canary_gold
-f /tmp/airflow_hiveop_ZixLMN/tmphgm9MR
{code}
(note that "Starting attempt X of Y has been removed from this log).

By default "1" would be opened if it exists when a user clicks into the log, otherwise dag-parsing
would be opened.


  was:
Ideally there would be tabs for logs:
Dag-parsing (needs a better name)
1 (Short for "Run 1")
2
...

Dag-parsing would contain the output like:
{code}
[2017-04-14 15:24:54,341] {models.py:168} INFO - Filling up the DagBag from /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,342] {models.py:253} DEBUG - Importing /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_r>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_large>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_tiny>
[2017-04-14 15:24:54,357] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_platinum_medium>
{code}

By default "1" would be opened if it exists when a user clicks into the log, otherwise dag-parsing
would be opened.




> Make logs easier to read and separate infra/task logs
> -----------------------------------------------------
>
>                 Key: AIRFLOW-1111
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1111
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Dan Davydov
>
> Ideally there would be tabs for logs:
> Dag-parsing (needs a better name)
> 1 (Short for "Run 1")
> 2
> ...
> Dag-parsing would contain the output like:
> {code}
> [2017-04-14 15:24:54,341] {models.py:168} INFO - Filling up the DagBag from /srv/data/airflow/airflow_canaries/canaries.py
> [2017-04-14 15:24:54,342] {models.py:253} DEBUG - Importing /srv/data/airflow/airflow_canaries/canaries.py
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_medium>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_r>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_large>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_medium>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_tiny>
> [2017-04-14 15:24:54,357] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_platinum_medium>
> [2017-04-14 15:24:54,553] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup memory
in / since it already exists
> [2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup airflow
in /memory since it already exists
> [2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup 2017-04-14
in /memory/airflow since it already exists
> [2017-04-14 15:24:54,554] {cgroup_task_runner.py:77} DEBUG - Creating cgroup 82d42ce2-2126-11e7-ad73-22000a896f7d
in /memory/airflow/2017-04-14
> {code}
> and "1" would contain
> {code}
> 2017-04-14 15:24:56,146] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,146]
{models.py:1344} INFO - Executing <Task(HiveOperator): insert_canary_row> on 2017-04-14
15:10:00
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,172]
{hive_operator.py:98} INFO - Executing: USE datainfra;
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask:           CREATE TABLE
IF NOT EXISTS datainfra.airflow_precious_canary_gold_large (key STRING);
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: INSERT OVERWRITE TABLE
datainfra.airflow_precious_canary_gold_large
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: SELECT key FROM datainfra.one_row
LIMIT 1;
> [2017-04-14 15:24:56,187] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,186]
{hive_hooks.py:207} INFO - hive -hiveconf airflow.ctx.dag.dag_id=airflow_precious_canary_gold_large
-hiveconf airflow.ctx.dag_run.execution_date=2017-04-14T15:10:00 -hiveconf airflow.ctx.task_instance.execution_date=2017-04-14T15:10:00
-hiveconf airflow.ctx.task.task_id=insert_canary_row -hiveconf mapreduce.job.queuename=airflow_canary_gold
-f /tmp/airflow_hiveop_ZixLMN/tmphgm9MR
> {code}
> (note that "Starting attempt X of Y has been removed from this log).
> By default "1" would be opened if it exists when a user clicks into the log, otherwise
dag-parsing would be opened.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message