From dev-return-91046-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Fri Jan 19 19:34:44 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id BDCF5180607 for ; Fri, 19 Jan 2018 19:34:44 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id AD43A160C36; Fri, 19 Jan 2018 18:34:44 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F3356160C1B for ; Fri, 19 Jan 2018 19:34:43 +0100 (CET) Received: (qmail 48954 invoked by uid 500); 19 Jan 2018 18:34:42 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Delivered-To: moderator for dev@kafka.apache.org Received: (qmail 137 invoked by uid 99); 19 Jan 2018 17:23:18 -0000 X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.715 X-Spam-Level: ** X-Spam-Status: No, score=2.715 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_IMAGE_ONLY_28=0.726, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=thoughtworks-com.20150623.gappssmtp.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thoughtworks-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=xcnE+6F5TVzusHKsqobBqnlHnRfXfjES7Rd9QGAog+U=; b=wJt3h2YANXM1b9MpflZA1QNTQ8/I2COtiq2KqXR2lHZTUWlOhO2ir1AayZtlWafOJg dBRaadufKbaYPooDQKrpSF7xjwhe0u197kqb/nRxOWFOxKnD+wljfUT2wzfcKC9D4fmh qVBovZv/UU8t4TI/8g3DMOMxW1RCnvQhrINHrEif6UbMC8WAdNlFEaQ2IREgEXO/GYD3 RSs0DxtNBfkNbrjDVRyXwrmfpfadWlMLy1kQ2Jztye008RvBvxrev4Sn8QFoxJGy7k5U NXvkK06V84gv6Nq07kNBaxR4s7XwRY72X83OS41Q2BfpaZGRaRQKTnRRHSAoH01NLeD1 MlUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=xcnE+6F5TVzusHKsqobBqnlHnRfXfjES7Rd9QGAog+U=; b=kWDmMjwMKCYTy6sQIwmHLQkimeJsfPIbVHDbcAQFopVO+5TSPjhXXZPeUc1d9zL/MP a3xt7rJus+4UoVWQANMXMwo6vIsXs+zgfGaTuu9R78zu/3qOcgC2FUxh1tohgEn4Knke pamZ6q5VAM5+ZWskUrI/XljQUNjGkUq7BQCvy3soKm6u2eTRGQLuA4y5MA066ZEIiTyw t0UYIg9Ilw9e1fyTJNafu02D27uwndamAgnX7rfvactHpZNlu1eTPLF53PF1GvOjRjaP bc0cKL7sRUhZ+GcXmLjY/a8HpJ5CxkVnMGMvLeaOeqlmNJBy4o1YnSifPVkJJbUwbLwS ZohA== X-Gm-Message-State: AKwxytfvuQXvFoMna/TS7M9o2mW9q88GXCr1uUakeEcUrgE5qcoUjhYd n+6EdVlfkcRwfldCz/6Vkl3LR05F2CgzxtnoeUrLzVRuAF8= X-Google-Smtp-Source: ACJfBovtl9bdm0ieoOJPj/9lmk9Hjuvx1e711i+zZs+RAaCzvWlos+BY8+/g2LUxMZPD282ZdXTcuTfMjPSrHhFTt1M= X-Received: by 10.13.235.65 with SMTP id u62mr9836368ywe.271.1516382588661; Fri, 19 Jan 2018 09:23:08 -0800 (PST) MIME-Version: 1.0 From: Saloni Vithalani Date: Fri, 19 Jan 2018 22:53:08 +0530 Message-ID: Subject: How many kafka streams app is recommended to run on single machine in production ? To: dev@kafka.apache.org, users@kafka.apache.org Content-Type: multipart/alternative; boundary="94eb2c0866c48a912f05632455f8" --94eb2c0866c48a912f05632455f8 Content-Type: text/plain; charset="UTF-8" In our architecture, we are assuming to run three jvm processes on one machine (approx.) and each jvm machine can host upto 15 kafka-stream apps. And if I am not wrong each kafka-stream app spawns one java thread. So, this seems like an awkward architecture to have with around 45 kafka-stream apps running on a single machine. So, I have question in three parts 1) Is my understanding correct that each kafka-stream app spawns one java thread ? Also, each kafka-stream starts a new tcp connection with kafka-broker ? 2) Is there a way to share one tcp connection for multiple kafka-streams ? 3) Is is difficult(not recommended) to run 45 streams on single machine ? The answer to this is definitely NO unless there is a real use case in production. Regards, Saloni Vithalani Developer Email saloniv@thoughtworks.com Telephone +91 8552889571 <8552889571> [image: ThoughtWorks] --94eb2c0866c48a912f05632455f8--