airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Park (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-610) Configuration parsing order doesn't work properly.
Date Wed, 02 Nov 2016 01:42:58 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627304#comment-15627304
] 

Yongjun Park edited comment on AIRFLOW-610 at 11/2/16 1:42 AM:
---------------------------------------------------------------

The exception which I attached occurred when I've made a change to the configuration to use
_sql_alchemy_conn_cmd_.
Originally it was _sql_alchemy_conn_ and I created databases with this option by _airflow
initdb_.

You can see this invalid operation with following steps.

# create an airflow.cfg file:
{code}
[core]
sql_alchemy_conn_cmd = /home/airflow/db_conn.sh
{code}
# _/home/airflow/db_conn.sh_ contains:
{code}
#!/bin/bash
printf mysql+mysqldb://airflow:airflow@localhost:3306/airflow
{code}
# you need to run _airflow initdb_
{noformat}
$ airflow initdb
[2016-11-01 09:54:18,041] {__init__.py:57} INFO - Using executor SequentialExecutor
[2016-11-01 09:54:18,114] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-11-01 09:54:18,132] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: sqlite:////home/airflow/airflow.db
[2016-11-01 09:54:18,382] {db.py:284} INFO - Creating tables
....
{noformat}

Although _sql_alchemy_conn_cmd_ is exists, airflow is trying to create db using sqlite.

I think default configuration precedes over __cmd_ configurations.
Your test needs to override default options instead of deleting them.


was (Author: yongjun):
The exception which I attached occurred when I've made a change to the configuration to use
_sql_alchemy_conn_cmd_.
Originally it was _sql_alchemy_conn_ and I created databases with this option by _airflow
initdb_.

You can see this invalid operation following steps.

# create an airflow.cfg file:
{code}
[core]
sql_alchemy_conn_cmd = /home/airflow/db_conn.sh
{code}
# _/home/airflow/db_conn.sh_ contains:
{code}
#!/bin/bash
printf mysql+mysqldb://airflow:airflow@localhost:3306/airflow
{code}
# you need to run _airflow initdb_
{noformat}
$ airflow initdb
[2016-11-01 09:54:18,041] {__init__.py:57} INFO - Using executor SequentialExecutor
[2016-11-01 09:54:18,114] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-11-01 09:54:18,132] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: sqlite:////home/airflow/airflow.db
[2016-11-01 09:54:18,382] {db.py:284} INFO - Creating tables
....
{noformat}

Although _sql_alchemy_conn_cmd_ is exists, airflow is trying to create db using sqlite.

I think default configuration precedes over __cmd_ configurations.
Your test needs to override default options instead of deleting them.

> Configuration parsing order doesn't work properly.
> --------------------------------------------------
>
>                 Key: AIRFLOW-610
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-610
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Yongjun Park
>            Assignee: Yongjun Park
>
> I'm testing Airflow using master branch, precisely 527e3ec.
> Documents says configuration is evaluated following order.
> # environment variable
> # configuration in airflow.cfg
> # command in airflow.cfg
> # default
> However, it can't recognize *_cmd* options.
> I added *sql_alchemy_conn_cmd* option. but it printed error below: 
> {code}
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 6, in <module>
>     exec(compile(open(__file__).read(), __file__, 'exec'))
>   File "/usr/local/airflow/airflow/bin/airflow", line 17, in <module>
>     from airflow import configuration
>   File "/home/airflow/airflow/__init__.py", line 30, in <module>
>     from airflow import configuration as conf
>   File "/home/airflow/airflow/configuration.py", line 733, in <module>
>     conf.read(AIRFLOW_CONFIG)
>   File "/home/airflow/airflow/configuration.py", line 585, in read
>     self._validate()
>   File "/home/airflow/airflow/configuration.py", line 497, in _validate
>     self.get('core', 'executor')))
> {code}
> I think it is affected by DEFAULT configuration. It should be changed evaluation order
as document's saying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message