From dev-return-6893-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Mon Oct 29 01:41:51 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 25385180671 for ; Mon, 29 Oct 2018 01:41:50 +0100 (CET) Received: (qmail 19675 invoked by uid 500); 29 Oct 2018 00:41:50 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 19663 invoked by uid 99); 29 Oct 2018 00:41:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Oct 2018 00:41:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0349018E433 for ; Mon, 29 Oct 2018 00:41:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.989 X-Spam-Level: * X-Spam-Status: No, score=1.989 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=smartnews-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 8t6Z6XEz7zGV for ; Mon, 29 Oct 2018 00:41:47 +0000 (UTC) Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id F18745F331 for ; Mon, 29 Oct 2018 00:41:46 +0000 (UTC) Received: by mail-oi1-f172.google.com with SMTP id k19-v6so5573940oic.11 for ; Sun, 28 Oct 2018 17:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartnews-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=vvfVSb4Uu7ulFs8TiL18S6VGpFrfDGLtM+Ysl9nBj7Q=; b=PbPb4weSBDy23HDr5UzBTe8QefiMDjslRCkYsrQvvaZfvaT/vxywVwB5LP1KyTZHVd Nj7vJ1ws9BCeHze9iWtwAbdiJaZNz9ZMkJ+OOi6HNGFnPanDUEmtKBPLQEQGY74VT/By 8+oSfa7300/iET4QS1Niyx/Q4IVD/HUya/GRoDhKFQHVMoQ3++i2tgmmhCY47bkmM/JH 1IvsFB4vzMniGQ3HU9kS0lUJeLqbCWpi/YpoaWoLGELOD748ZYYG86Rv/EUfiKySEyut LjH0Yw0jJilpRPP7GdtPF+eJPXujk2fRbW7h+F4is0avhKvK3uK8QJ2xOIq3Wg+ETuhW vN0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=vvfVSb4Uu7ulFs8TiL18S6VGpFrfDGLtM+Ysl9nBj7Q=; b=r/K6zTLHxsEJ+ugiAE7pZegLXVcnNu6VDgdi0n7BfIKAttoPjZNTKER1pCFxo2lMpA og3I9QxDRpr1zTfosRbuiKXn5WjIpKBvD6bbrmY/gW5Kc798Mu+8obgcVA6KUBsJklaN c6iQsIghHPgncp2rCCSIfcFuYHdyoA7MLSy5JmNsXMAFmunW2BsQoa9PiSr5E8azUqtr CV2k/gak+wAI7SHVOz+4JWbrnRLZ/TXVdamnQDZMDlJH7tv6ytoO6nSu1gJMdC1ZUe74 JCC5+ricxXud7BO3suKGtiP0bjopOzi7an84zSQVpzActt6lAzp4fDzWT8ve5dW3pI1i L8Fg== X-Gm-Message-State: AGRZ1gK2igtTqh+u+j9+FedXolDD4paHo0CUz4cgyWtXEYYVtiOv96E3 Gaj7z2pEgZFvF90Gui0++oKwyRd2MUMqqZBrV4vOLCY= X-Google-Smtp-Source: AJdET5f6d2hNk1XZNIAuekjRgL7atytURRx4KZ1DCZ8enrBTmL6Tx2wEzDQfOSiHSplJ+0aaCD4H+mdQMry8sw5OVlE= X-Received: by 2002:aca:3c56:: with SMTP id j83-v6mr7614475oia.155.1540773705358; Sun, 28 Oct 2018 17:41:45 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jerry Chi Date: Mon, 29 Oct 2018 09:41:33 +0900 Message-ID: Subject: Re: Guidelines around how to scale with worker nodes? To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000633a110579535580" --000000000000633a110579535580 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sorry, any tips or hints related the below questions? Thank you. Jerry 2018=E5=B9=B410=E6=9C=8824=E6=97=A5(=E6=B0=B4) 3:37=E3=80=81Jerry Chi =E3= =81=95=E3=82=93=EF=BC=88jerry.chi@smartnews.com=EF=BC=89=E3=81=AE=E3=83=A1= =E3=83=83=E3=82=BB=E3=83=BC=E3=82=B8: > Hi everyone, > > I'm occasionally observing tasks stuck in "queued" for a long time despit= e > trying various edits of parameter values in airflow.cfg and I'm guessing = it > would help to increase the number of worker nodes (right now I have one > worker node). > > Are there any guidelines for: > 1. How to determine if the # of worker nodes is indeed the bottleneck > causing tasks to be stuck in "queued" ? It doesn't seem the memory/CPU > usage on the worker node is close to 100%. > 2. How to determine the optimal number and CPU/memory specs of the worker > nodes if I want to be able to handle X simultaneous tasks without them > getting stuck in "queued" ? > I'm using CeleryExecutor + RabbitMQ on EC2. > > Thanks~ > Jerry Chi =E3=82=B8=E3=82=A7=E3=83=AA=E3=83=BC=E3=83=BB=E3=83=81=E3=83=BC= | Data Science Manager | +81-70-2668-5491 | LINE/Skype: > peacej | =EC=B9=B4=ED=86=A1: peacej2 | WeChat: jerrychijerry > --000000000000633a110579535580--