airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirudh Ramanathan <ramanath...@google.com.INVALID>
Subject Re: Airflow + Kubernetes + Git Sync
Date Sat, 13 Jan 2018 00:17:54 GMT
> Any good way to debug this?

One way might be reading the events from "kubectl get events". That should
reveal some information about the pod removal event.
This brings up another question - should errored pods be persisted for
debugging?

On Fri, Jan 12, 2018 at 3:07 PM, jordan.zucker@gmail.com <
jordan.zucker@gmail.com> wrote:

> I'm trying to use Airflow and Kubernetes and having trouble using git sync
> to pull DAGs into workers.
>
> I use a git sync init container on the scheduler to pull in DAGs initially
> and that works. But when worker pods are spawned, the workers terminate
> almost immediately because they cannot find the DAGs. But since the workers
> terminate so quickly, I can't even inspect the file structure to see where
> the DAGs ended up during the workers git sync init container.
>
> I noticed that the git sync init container for the workers is hard coded
> into /tmp/dags and there is a git_subpath config setting as well. But I
> can't understand how the git synced DAGs ever end up in /root/airflow/dags
>
> I am successfully using a git sync init container for the scheduler, so I
> know my git credentials are valid. Any good way to debug this? Or an
> example of how to set this up correctly?
>



-- 
Anirudh Ramanathan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message