airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Davies (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-2741) Example Kubernetes pod operator doesn't work
Date Tue, 10 Jul 2018 13:11:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jon Davies updated AIRFLOW-2741:
--------------------------------
    Description: 
I'm trying to make the DAG from this blog post work:
 
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/

But it fails with:

{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375] {kubernetes_executor.py:579}
INFO - self.running: {(u'kubernetes_sample', u'run_this_first', datetime.datetime(2018, 7,
10, 13, 0, 17, 332338, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow
run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: + airflow
run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:57,667] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5,
pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:58,023] {models.py:257} INFO - Filling up the DagBag from /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:58,029] {models.py:348} INFO - File /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py
assumed to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: Traceback
(most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/bin/airflow",
line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py",
line 74, in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     return
f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py",
line 475, in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     dag
= get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py",
line 146, in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: airflow.exceptions.AirflowException:
dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to
parse.
{code}

...note that the DAG isn't in my Docker image, I just cp'ed it to the webserver/scheduler
as the blog post did in "Step 5: Upload a test document".

  was:
I'm trying to make the DAG from this blog post work:
 
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/

But it fails with:

{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375] {kubernetes_executor.py:579}
INFO - self.running: {(u'kubernetes_sample', u'run_this_first', datetime.datetime(2018, 7,
10, 13, 0, 17, 332338, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow
run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: + airflow
run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:57,667] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5,
pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:58,023] {models.py:257} INFO - Filling up the DagBag from /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:58,029] {models.py:348} INFO - File /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py
assumed to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: Traceback
(most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/bin/airflow",
line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py",
line 74, in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     return
f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py",
line 475, in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     dag
= get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py",
line 146, in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:     'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: airflow.exceptions.AirflowException:
dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to
parse.
{code}

...note that the DAG isn't in my Docker image, I just cp'ed it to the webserver/scheduler
as the blog post did.


> Example Kubernetes pod operator doesn't work
> --------------------------------------------
>
>                 Key: AIRFLOW-2741
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2741
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Jon Davies
>            Priority: Major
>
> I'm trying to make the DAG from this blog post work:
>  
> https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
> But it fails with:
> {code}
> airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375] {kubernetes_executor.py:579}
INFO - self.running: {(u'kubernetes_sample', u'run_this_first', datetime.datetime(2018, 7,
10, 13, 0, 17, 332338, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow
run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags'}
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: + airflow
run kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:57,667] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5,
pool_recycle=1800
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:58,023] {models.py:257} INFO - Filling up the DagBag from /root/airflow/dags
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: [2018-07-10
13:00:58,029] {models.py:348} INFO - File /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py
assumed to contain no DAGs. Skipping.
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: Traceback
(most recent call last):
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File
"/usr/local/bin/airflow", line 32, in <module>
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   
 args.func(args)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File
"/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in wrapper
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   
 return f(*args, **kwargs)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File
"/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475, in run
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   
 dag = get_dag(args)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   File
"/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146, in get_dag
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:   
 'parse.'.format(args.dag_id))
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]: airflow.exceptions.AirflowException:
dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to
parse.
> {code}
> ...note that the DAG isn't in my Docker image, I just cp'ed it to the webserver/scheduler
as the blog post did in "Step 5: Upload a test document".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message