airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes
Date Sat, 28 Apr 2018 18:29:33 GMT
Also one of the main benefits of the Kubernetes Executor is having a Docker
image that contains all the dependencies that you need for your job.
Personally I would switch to Kubernetes when it leaves the experimental
stage.

Cheers, Fokko

2018-04-28 16:27 GMT+02:00 Kyle Hamlin <hamlin.kn@gmail.com>:

> I don't have a Dask cluster yet, but I'm interested in taking advantage of
> it for ML tasks. My use case would be bursting a lot of ML jobs into a
> Dask cluster all at once.
> From what I understand, Dask clusters utilize caching to help speed up jobs
> so I don't know if it makes sense to launch a Dask cluster for every single
> ML job. Conceivably, I could just have a single Dask worker running 24/7
> and when its time to burst k8 could autoscale the Dask workers as more ML
> jobs are launched into the Dask cluster?
>
> On Fri, Apr 27, 2018 at 10:35 PM Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> > Hi Kyle,
> >
> > So you have a static Dask cluster running your k8s cluster? Is there any
> > reason you wouldn't just launch the Dask cluster for the job you're
> running
> > and then tear it down? I feel like with k8s the elasticity is one of the
> > main benefits.
> >
> > On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin <hamlin.kn@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > If I have a Kubernetes cluster running in DCOC and a Dask cluster
> running
> > > in that same Kubernetes cluster is it possible/does it makes sense to
> use
> > > the KubernetesExecutor to launch tasks into the Dask cluster (these are
> > ML
> > > jobs with sklearn)? I feel like there is a bit of inception going on
> here
> > > in my mind and I just want to make sure a setup like this makes sense?
> > > Thanks in advance for anyone's input!
> > >
> >
>
>
> --
> Kyle Hamlin
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message