From dev-return-4893-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Wed Apr 18 08:19:44 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8989418064E for ; Wed, 18 Apr 2018 08:19:43 +0200 (CEST) Received: (qmail 50322 invoked by uid 500); 18 Apr 2018 06:19:42 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Delivered-To: moderator for dev@airflow.incubator.apache.org Received: (qmail 38324 invoked by uid 99); 18 Apr 2018 06:15:06 -0000 Date: Wed, 18 Apr 2018 06:15:04 -0000 Message-ID: x-ponymail-agent: PonyMail Composer/0.3 To: x-ponymail-sender: e2465d68e847b4802529b215ea360f6e344f8baf In-Reply-To: Subject: Re: Benchmarking of Airflow Scheduler with Celery Executor From: yrqls21@gmail.com References: MIME-Version: 1.0 X-Mailer: LuaSocket 3.0-rc1 Content-Type: text/plain; charset=utf-8 On 2018/04/13 17:00:36, Maxime Beauchemin wrote: > If you're concerned about scheduler scalability I'd go with a bigger box. > The scheduler uses multiprocessing so more CPU power means more throughput. > > Also you may want to provision a beefy MySQL box to make sure that doesn't > become the bottleneck. 10k tasks heartbeating to the DB every 30 seconds is > significant load. > > Perhaps Airbnb folks chime in about their scale and hardware setup? > > Max > > On Fri, Apr 13, 2018 at 9:14 AM, ramandumcs@gmail.com > wrote: > > > Thanks Ry, > > Just wondering if there is any approximate number on concurrent tasks a > > scheduler can run on say 16 GB RAM and 8 core machine. > > If its already been done that would be useful. > > We did some benchmarking with local executor and observed that each > > TaskInstance was taking ~100MB of memory so we could only run ~130 > > concurrent tasks on 16 GB RAM and 8 core machine. > > > > -Raman Gupta > > > > > > > > On 2018/04/12 16:32:37, Ry Walker wrote: > > > Hi Raman - > > > > > > First, we’d be happy to help you test this out with Airflow. Or you could > > > do it yourself by using http://open.astronomer.io/airflow/ (w/ Docker > > > Engine + Docker Compose) to quickly spin up a test environment. > > Everything > > > is hooked to Prometheus/Grafana to monitor how the system reacts to your > > > workload. > > > > > > -Ry > > > CEO, Astronomer > > > > > > On April 12, 2018 at 12:23:46 PM, ramandumcs@gmail.com ( > > ramandumcs@gmail.com) > > > wrote: > > > > > > Hi All, > > > We have requirement to run 10k(s) of concurrent tasks. We are exploring > > > Airflow's Celery Executor for same. Horizontally Scaling of worker nodes > > > seem possible but it can only have one active scheduler. > > > So will Airflow scheduler be able to handle these many concurrent tasks. > > > Is there any benchmarking number around airflow scheduler's scalability. > > > Thanks, > > > Raman > > > > > > With an AWS EC2 i2.8xlarge box, we run ~14k tasks at peek. Though the scheduling delay also spikes to ~30 mins when we are at peek load. Here's some scheduler config we have: JOB_HEARTBEAT_SEC = 60 MAX_THREADS = 64 MAX_TIS_PER_QUERY = 512 Also we have the biggest Amazon RDS mysql instance. Cheers, Kevin Yang