airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Standish <dpstand...@gmail.com>
Subject Re: Backing up PostgreSQL DB on K8s
Date Thu, 10 Jun 2021 01:01:23 GMT
> Thanks Daniel. We would need to add this as an option to the K8s Helm
Chart and that is kinda outside our current scope.

I may be misunderstanding what you're saying, but there's no need to modify
the helm chart.

Generally speaking all airflow helm charts support using a metastore that
is deployed outside of kubernetes.

Airflow just needs the creds.

E.g. with the official helm chart, you can use an external database with
this configuration (i.e. in helm config.yaml):

data:
  metadataSecretName: metastore-uri
  resultBackendSecretName: result-backend

pgbouncer:
  enabled: false

postgresql:
  enabled: false

All that is required is you create k8s secrets with the conn URI.

The secrets have to be structured like this:

locals {
  metastore_uri = ""
}
resource "kubernetes_secret" "metastore" {
  metadata {
    name = "metastore-uri"
    namespace = var.environment_name
  }
  data = {
    connection = "postgresql://${local.metastore_uri}"
  }
  type = "Opaque"
}

resource "kubernetes_secret" "result_backend" {
  metadata {
    name = "result-backend"
    namespace = var.environment_name
  }
  data = {
    connection = "db+${kubernetes_secret.metastore.data.connection}"
  }
  type = "Opaque"
}









On Wed, Jun 9, 2021 at 4:04 PM Lewis John McGibbney <lewismc@apache.org>
wrote:

> Thanks Daniel. We would need to add this as an option to the K8s Helm
> Chart and that is kinda outside our current scope.
> Thanks for the suggestion.
> lewismc
>
> On 2021/06/09 05:46:00, Daniel Standish <dpstandish@gmail.com> wrote:
> > Perhaps it goes without saying but you might consider using cloud sql
> > option such as aws rds, which provides persistence even if you destroy
> and
> > rebuild your k8s cluster, and of course automated backups.
> >
> > On Tue, Jun 8, 2021, 10:41 PM Sumit Maheshwari <msumit@apache.org>
> wrote:
> >
> > > If you are backing up data to safeguard against pod failures, then I
> > > believe that you can use a PV as data storage for PSql & it would
> survive
> > > any pod restarts.
> > >
> > > On Wed, Jun 9, 2021 at 5:01 AM Lewis John McGibbney <
> lewismc@apache.org>
> > > wrote:
> > >
> > >> Hi users@,
> > >> Does anyone have a recommended/best practice/preferred way for
> backing up
> > >> PostgreSQL when Airflow is deployed into K8s?
> > >> We were thinking of writing a maintenance DAG which would do this...
> > >> maybe even contributing it to
> > >> https://github.com/teamclairvoyant/airflow-maintenance-dags.
> > >> I'm thinking it would just authenticate into the K8s cluster, find the
> > >> postgresql pod and then perform a DB archival in s3 or something.
> > >> I also looked more into
> > >> https://www.postgresql.org/docs/9.1/continuous-archiving.html which
> > >> looks appealing. I'm interest to see what others are doing.
> > >> Any suggestions appreciated.
> > >> Thank you
> > >> lewismc
> > >>
> > >
> >
>

Mime
View raw message