airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John McGibbney <lewi...@apache.org>
Subject Re: Backing up PostgreSQL DB on K8s
Date Thu, 10 Jun 2021 14:54:59 GMT
wow OK I didn't know this was an option.
I'll see what I can do and write back if we make progress.
Thank you Daniel.

On 2021/06/10 01:01:23, Daniel Standish <dpstandish@gmail.com> wrote: 
> > Thanks Daniel. We would need to add this as an option to the K8s Helm
> Chart and that is kinda outside our current scope.
> 
> I may be misunderstanding what you're saying, but there's no need to modify
> the helm chart.
> 
> Generally speaking all airflow helm charts support using a metastore that
> is deployed outside of kubernetes.
> 
> Airflow just needs the creds.
> 
> E.g. with the official helm chart, you can use an external database with
> this configuration (i.e. in helm config.yaml):
> 
> data:
>   metadataSecretName: metastore-uri
>   resultBackendSecretName: result-backend
> 
> pgbouncer:
>   enabled: false
> 
> postgresql:
>   enabled: false
> 
> All that is required is you create k8s secrets with the conn URI.
> 
> The secrets have to be structured like this:
> 
> locals {
>   metastore_uri = ""
> }
> resource "kubernetes_secret" "metastore" {
>   metadata {
>     name = "metastore-uri"
>     namespace = var.environment_name
>   }
>   data = {
>     connection = "postgresql://${local.metastore_uri}"
>   }
>   type = "Opaque"
> }
> 
> resource "kubernetes_secret" "result_backend" {
>   metadata {
>     name = "result-backend"
>     namespace = var.environment_name
>   }
>   data = {
>     connection = "db+${kubernetes_secret.metastore.data.connection}"
>   }
>   type = "Opaque"
> }
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Wed, Jun 9, 2021 at 4:04 PM Lewis John McGibbney <lewismc@apache.org>
> wrote:
> 
> > Thanks Daniel. We would need to add this as an option to the K8s Helm
> > Chart and that is kinda outside our current scope.
> > Thanks for the suggestion.
> > lewismc
> >
> > On 2021/06/09 05:46:00, Daniel Standish <dpstandish@gmail.com> wrote:
> > > Perhaps it goes without saying but you might consider using cloud sql
> > > option such as aws rds, which provides persistence even if you destroy
> > and
> > > rebuild your k8s cluster, and of course automated backups.
> > >
> > > On Tue, Jun 8, 2021, 10:41 PM Sumit Maheshwari <msumit@apache.org>
> > wrote:
> > >
> > > > If you are backing up data to safeguard against pod failures, then I
> > > > believe that you can use a PV as data storage for PSql & it would
> > survive
> > > > any pod restarts.
> > > >
> > > > On Wed, Jun 9, 2021 at 5:01 AM Lewis John McGibbney <
> > lewismc@apache.org>
> > > > wrote:
> > > >
> > > >> Hi users@,
> > > >> Does anyone have a recommended/best practice/preferred way for
> > backing up
> > > >> PostgreSQL when Airflow is deployed into K8s?
> > > >> We were thinking of writing a maintenance DAG which would do this...
> > > >> maybe even contributing it to
> > > >> https://github.com/teamclairvoyant/airflow-maintenance-dags.
> > > >> I'm thinking it would just authenticate into the K8s cluster, find
the
> > > >> postgresql pod and then perform a DB archival in s3 or something.
> > > >> I also looked more into
> > > >> https://www.postgresql.org/docs/9.1/continuous-archiving.html which
> > > >> looks appealing. I'm interest to see what others are doing.
> > > >> Any suggestions appreciated.
> > > >> Thank you
> > > >> lewismc
> > > >>
> > > >
> > >
> >
> 

Mime
View raw message