airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maulik Soneji <maulik.son...@gojek.com>
Subject Re: [Question] [Airflow on Kubernetes] Syncing airflow variables from file
Date Wed, 03 Jun 2020 12:03:59 GMT
@Kamil Thank you very much for working on this PR. This certainly helps our
case where we need to load the variables from a file. I will work on
testing this feature.

@Ash Thank you very much for your inputs. We are currently using
KubernetesExecutor along with airflow_local_settings.py(mounted as a
configmap) to add labels to the pods created by KubernetesExecutor. With
the current approach, we have to modify the configmap as well to send the
environment variables to the created pod. I think to go with file-based
approach would be better since we are currently syncing the dags and
variables file. The environment variable based approach would require us to
redeploy with the updated deployment configuration.

Thanks and regards,
Maulik

On Wed, Jun 3, 2020 at 4:50 PM Ash Berlin-Taylor <ash@apache.org> wrote:

> Or also, depending on your use case, you could use
> https://github.com/apache/airflow/pull/7923/ to get Variables from env
> vars prefixed with AIRFLOW_VAR_
>
> This one is already included in 1.10.10. It's not quite what you asked
> for, but if you can feed those variables in to your Kube pod definitions
> it would work now.
>
> -ash
>
>
> On Jun 3 2020, at 11:39 am, Kamil BreguĊ‚a <kamil.bregula@polidea.com>
> wrote:
>
> > You can use Local Filesystem Secret Backend. This will allow you to
> > read the variables directly from the file.
> > https://github.com/apache/airflow/pull/8436
> > It has not been released yet, but you can now copy the code and start
> > testing it. This will allow us to release a new version with your
> > suggestions if you have any.
> >
> > Best regards,
> > Kamil
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message