From dev-return-4959-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Sat Apr 28 20:29:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3D2C9180649 for ; Sat, 28 Apr 2018 20:29:46 +0200 (CEST) Received: (qmail 88291 invoked by uid 500); 28 Apr 2018 18:29:45 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 88275 invoked by uid 99); 28 Apr 2018 18:29:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Apr 2018 18:29:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E433A180070 for ; Sat, 28 Apr 2018 18:29:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.229 X-Spam-Level: ** X-Spam-Status: No, score=2.229 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=driesprongen-nl.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id v8Mu7ToMIZfV for ; Sat, 28 Apr 2018 18:29:41 +0000 (UTC) Received: from mail-yb0-f173.google.com (mail-yb0-f173.google.com [209.85.213.173]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id CD8B85F188 for ; Sat, 28 Apr 2018 18:29:40 +0000 (UTC) Received: by mail-yb0-f173.google.com with SMTP id q74-v6so1823309ybg.6 for ; Sat, 28 Apr 2018 11:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=driesprongen-nl.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=KLVTINY9VX185S2ww9FNF54uejSQUeyvHwKRB4P+/bk=; b=OYdzJRM3WCaN1B2w/HDwchRvYgLAxvqrJisvJuASeOwoUx+uwO/Ic1g/Jm8mlbt79P UhEt6oEr3T9t8aWr8Ms8DBIfRjhTLT20YpRhm00AuhLtM6KeZY3CosA8/XRX2Ro1Wlby hBNMQHRDocE7VT8Wm1FnxPPkZBYLF/W7qMrMOLNhFI1Zks9t9Jb73s9XDrrzyx+ubEvE a4SgxcPYPCJqcGGq8EfwafumBesZKFV2sLfMlq6nqx0PP+YE33bBr/lzAhzBzlUC1RXR waU2LvTcZhhbWI4OIFg7OHX0uOBNwvHreOe5vcAexukdzgpk0Bz4Qjm35ouzs+U7N0F2 S9Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=KLVTINY9VX185S2ww9FNF54uejSQUeyvHwKRB4P+/bk=; b=nMOZjqfeuSUUgbyGp6v6SMgxo5t7eBFSYmteCor2mGVqY04faFGWyt9NIoJL3YHBLf 13a+Wbg9Qpeqg0PB5RZmeft+McQOIN8+QUGajM3F6bTWurT6u6BWo99/6W/C1I/tLJhs 2Aa3ynaDnBsyzP8C0ZinMiBdSwztupTv9h+vA4bPOmccYxIBebWEqFA/VCQKibci6zh1 /t4aRtOLbGBR602GIxOf42nz+yMCC+LzRfVIKbXT8Fv101wFyBNEwArObz3SdHnYlFBV p5obHp5cq/y5dRGCIVlin1Kd5FlFaHZIEddE5F2ZTLgzIyIz8sxcFUSOMgG3yRB/6Xiw O/8g== X-Gm-Message-State: ALQs6tCKLdmJUX5OEHNMhuS7Jv4w0JVPe6FYnzG6wH194wQPwgqsgDew CIXmBzIkGKLbXBvVpRfs+WFaGwQ/jtQx4Tzt0YFSIQ== X-Google-Smtp-Source: AB8JxZrtbyU+igHIr15c6juVOfBNZp8OktPjdAX/yFO0HTyVL+BsSaTrxASNVO7xgdJo3xMaoapCwfq+nS/UyH6to5c= X-Received: by 2002:a25:3bc5:: with SMTP id i188-v6mr1153151yba.352.1524940173707; Sat, 28 Apr 2018 11:29:33 -0700 (PDT) MIME-Version: 1.0 Sender: fokko@driesprongen.nl Received: by 2002:a25:67d5:0:0:0:0:0 with HTTP; Sat, 28 Apr 2018 11:29:33 -0700 (PDT) In-Reply-To: References: From: "Driesprong, Fokko" Date: Sat, 28 Apr 2018 20:29:33 +0200 X-Google-Sender-Auth: DS-yfpM0VqkYxhX88_dP4v7ZmkA Message-ID: Subject: Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="0000000000005ba434056aeccddf" --0000000000005ba434056aeccddf Content-Type: text/plain; charset="UTF-8" Also one of the main benefits of the Kubernetes Executor is having a Docker image that contains all the dependencies that you need for your job. Personally I would switch to Kubernetes when it leaves the experimental stage. Cheers, Fokko 2018-04-28 16:27 GMT+02:00 Kyle Hamlin : > I don't have a Dask cluster yet, but I'm interested in taking advantage of > it for ML tasks. My use case would be bursting a lot of ML jobs into a > Dask cluster all at once. > From what I understand, Dask clusters utilize caching to help speed up jobs > so I don't know if it makes sense to launch a Dask cluster for every single > ML job. Conceivably, I could just have a single Dask worker running 24/7 > and when its time to burst k8 could autoscale the Dask workers as more ML > jobs are launched into the Dask cluster? > > On Fri, Apr 27, 2018 at 10:35 PM Daniel Imberman < > daniel.imberman@gmail.com> > wrote: > > > Hi Kyle, > > > > So you have a static Dask cluster running your k8s cluster? Is there any > > reason you wouldn't just launch the Dask cluster for the job you're > running > > and then tear it down? I feel like with k8s the elasticity is one of the > > main benefits. > > > > On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin > wrote: > > > > > Hi all, > > > > > > If I have a Kubernetes cluster running in DCOC and a Dask cluster > running > > > in that same Kubernetes cluster is it possible/does it makes sense to > use > > > the KubernetesExecutor to launch tasks into the Dask cluster (these are > > ML > > > jobs with sklearn)? I feel like there is a bit of inception going on > here > > > in my mind and I just want to make sure a setup like this makes sense? > > > Thanks in advance for anyone's input! > > > > > > > > -- > Kyle Hamlin > --0000000000005ba434056aeccddf--