airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marty Woodlee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1026) connection string using _cmd tin airflow.cfg is broken
Date Wed, 29 Mar 2017 21:16:41 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947910#comment-15947910
] 

Marty Woodlee commented on AIRFLOW-1026:
----------------------------------------

I am having the same issue.

I believe this is happening because the Python {{ConfigParser}} behaves in an additive/"merging"
fashion when multiple configs are read, and at https://github.com/apache/incubator-airflow/blob/1.8.0/airflow/configuration.py#L524,
the built-in {{DEFAULT_CONFIG}} is read before the user's _actual_ config is loaded up at
https://github.com/apache/incubator-airflow/blob/1.8.0/airflow/configuration.py#L784. Since
the default config _does_ contain a {{sql_alchemy_conn}} entry (which points to a local SQLite
DB), the config code never tries to evaluate {{sql_alchemy_conn_cmd}} since doing so takes
lower precedence (see https://github.com/apache/incubator-airflow/blob/1.8.0/airflow/configuration.py#L592-L605).
At first glance I'm not sure any of the {{\_cmd}} options have _ever_ worked?

My first instinct is that probably the {{DEFAULT_CONFIG}} should only ever get read if it
is known that there is no user-supplied config yet. May need thought, though, if anyone is
actually reliant upon that "fall-back" behavior? (Let's hope not.)

> connection string using _cmd tin airflow.cfg is broken
> ------------------------------------------------------
>
>                 Key: AIRFLOW-1026
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1026
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: configuration
>    Affects Versions: Airflow 1.8
>            Reporter: Harish Singh
>            Priority: Critical
>             Fix For: Airflow 1.8
>
>
> sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py
> I am expectiing configure.py to be invoked.
> But it just throws:
>              "cannot use sqlite with the LocalExecutor"
> The connection string that my script "configure.py" would return is something like this:
> mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname
> But after debugging, I found that, my script is not getting invoked at all.
> This is my airflow.cfg:
> executor = LocalExecutor
> sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py 
> sql_alchemy_pool_size = 5
> sql_alchemy_pool_recycle = 3600
> I tried not using the script and directly hardcoding the conn_url
> sql_alchemy_conn = mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname
> It works.
> But  there is a regression bug if somebody wants to use "sql_alchemy_conn_cmd"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message