From user-return-14371-archive-asf-public=cust-asf.ponee.io@storm.apache.org Tue Apr 9 21:25:40 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 271F9180629 for ; Tue, 9 Apr 2019 23:25:40 +0200 (CEST) Received: (qmail 62435 invoked by uid 500); 9 Apr 2019 21:06:01 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 62425 invoked by uid 99); 9 Apr 2019 21:06:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Apr 2019 21:06:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 1A031C005E for ; Tue, 9 Apr 2019 21:25:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.049 X-Spam-Level: ** X-Spam-Status: No, score=2.049 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id tm2LCQdo4l51 for ; Tue, 9 Apr 2019 21:25:35 +0000 (UTC) Received: from mail-it1-f175.google.com (mail-it1-f175.google.com [209.85.166.175]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8ECCE5FD9F for ; Tue, 9 Apr 2019 21:25:35 +0000 (UTC) Received: by mail-it1-f175.google.com with SMTP id u65so7510499itc.2 for ; Tue, 09 Apr 2019 14:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=qZ2LLuesWNK6QHh+07noiZJOWXXdvBPQfIWv6zBbFPc=; b=GXgGCOqmNuQ8Q223HuTeSLJKUR4LxVJFb6kzpTy2WRqgwEmC4UY7t7Q7qBYxXFqUgp WSzB0RYECHI1znHpWqRgmpo7AP14a54Maoh+f24b5ZOqz7UWAPqfI4EKZmfJpHU/1YsI 5XWVigctAtJVGLRyr/dqjfP0xahiRwoIH9CceLMOq1eSu3C7pfh0vQDNOu272sKfmpfJ /1imHiooVTOU7xhFELnXjuc0lrVK0WyM2OAjEMr+URhik2TWlzF3OAcmIM2e9AYDH2Rl 4Ii3zFUtnqIaVk1KdEO8y2l+JV4IgOvGtq2WBkbTYWPJJ5Asu0V08hMfgPQetU/WoxiZ /EJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=qZ2LLuesWNK6QHh+07noiZJOWXXdvBPQfIWv6zBbFPc=; b=g4ovI+Bc6X4We6T/bDGkGiF5m3/vKI5FSzcPjQMoZ15wdBRwtVMwZHzQ5lVGgR9HNM yZHQQu2s/fiO+51hXkYrvSWGC5vc/pMySqK9R2KVOBlRLwXWmwvWFqtz18V6imOW9gBV 9Md3Amgh6Llxcr45j5AveVGkR+rpWJZFuT+sofOSD/bbzUJa7MRAGUgFeGIKe9PyfvYT o3FBBzZeT659ysoeG3oybi6725k8Coa0h1E6FQ86TqqWCBWlCChIxiWJPocw8cSJ4uQ7 R9Jo+w/5RhKFgN8eSKt5CJxlGsrfovD+J750kcL9QGftUYlGVs9dvSWe+4LJ/pCjAV0R Ac9Q== X-Gm-Message-State: APjAAAWoo9YldivQ+rfau4Nje3CC7JGmE+wZz/DuXoCvlh9PDiGGaG2J fA+qpIqTAj/mtHT3CsVYcqnBZUg34PbfQETHa9A9mrOG X-Google-Smtp-Source: APXvYqwzwnGIfPSihRqiM8noL0ZmvdNiTvNxvNnMmBZguvaCMZEaxe9/x3CCOUW+aiFRSSC4sOAsFM+XYxlAt9w9R/8= X-Received: by 2002:a24:50d5:: with SMTP id m204mr413746itb.103.1554845134744; Tue, 09 Apr 2019 14:25:34 -0700 (PDT) MIME-Version: 1.0 From: Jayant Sharma Date: Wed, 10 Apr 2019 02:55:24 +0530 Message-ID: Subject: Computing available parallelism units for storm executors To: user@storm.apache.org Content-Type: multipart/alternative; boundary="000000000000eff18f05861f970e" --000000000000eff18f05861f970e Content-Type: text/plain; charset="UTF-8" Hi, I was going through P. Taylor Goetz's talk on scaling storm cluster (2014). He made an interesting point of computing available parallelism units in a cluster. For CPU bound applications this limit was 1 executor/CPU core. I just wanted to be sure my understanding about this is clear: If I have 3 supervisor machines, each having 5 workers or JVMs and each machine has 16 CPU cores. I have 3*16 = 48 parallelism units to distribute among all of my topologies? Which means the sum of all spouts and bolts executors across all the topologies should be 48. What are the implications if I keep my executors more or less than this value? If my understanding is incorrect, can someone please explain how to compute parallelism units and relate them to number of executors. Thanks, Jayant Sharma --000000000000eff18f05861f970e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I was going th= rough=C2=A0P. Taylor Goetz's talk on scaling storm cluster (2014). He m= ade an interesting point of computing available parallelism units in a clus= ter. For CPU bound applications this limit was 1 executor/CPU core. I just = wanted to be sure my understanding about this is clear:

If I have 3 supervisor machines, each having 5 workers or JVMs and ea= ch machine has 16 CPU cores. I have 3*16 =3D 48 parallelism units to distri= bute among all of my topologies? Which means the sum of all spouts and bolt= s executors across all the topologies should be 48. What are the implicatio= ns if I keep my executors more or less than this value?

If my understanding is incorrect, can someone please explain how to c= ompute parallelism units and relate them to number of executors.
=
Thanks,
Jayant Sharma
--000000000000eff18f05861f970e--