From user-zh-return-2630-archive-asf-public=cust-asf.ponee.io@flink.apache.org Fri Mar 27 10:00:51 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 07635180637 for ; Fri, 27 Mar 2020 11:00:50 +0100 (CET) Received: (qmail 57113 invoked by uid 500); 27 Mar 2020 10:00:46 -0000 Mailing-List: contact user-zh-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-zh@flink.apache.org Delivered-To: mailing list user-zh@flink.apache.org Received: (qmail 56829 invoked by uid 99); 27 Mar 2020 10:00:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Mar 2020 10:00:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 49043C069C; Fri, 27 Mar 2020 10:00:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.2 X-Spam-Level: X-Spam-Status: No, score=-0.2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id QOD8eVEI-rhm; Fri, 27 Mar 2020 10:00:43 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::243; helo=mail-lj1-x243.google.com; envelope-from=karmagyz@gmail.com; receiver= Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 644717E177; Fri, 27 Mar 2020 10:00:43 +0000 (UTC) Received: by mail-lj1-x243.google.com with SMTP id w1so9532504ljh.5; Fri, 27 Mar 2020 03:00:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Qv8LTjW5UPN+tZLWTlEHVOjur6Z8x323OO+I8boeW5Y=; b=afCBrXiRADMQnvpTLYWCoNCpZJe2oR0tfyreB9Bak9QZucd0TjaOpmQ++2ZjMtKiA0 MFKH1zK9xP67qt04e9pt/2rfbnd/eXM5pFQpVEAwlLySsC/qrmJIihdUCHq/dRAlHAXK rFJtNVta/dwu8AGgXeFkaWGvk5ngLodZmHnNcalxrlGo8ZAXFapZcDw+l0/d1+mPTWZQ mgIBJkQxEHBQCikD8kbcBQHxkKQzGSx79Iny9zcjDDnkERygWdZh3l03u++DhDK18XKq 1zkleF+kdDTFTPDWlON/1gpjiEBHrxLj0+pph9jQW0CTcOtidNpz2Od9mdXPmbxx1PFC UCzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Qv8LTjW5UPN+tZLWTlEHVOjur6Z8x323OO+I8boeW5Y=; b=JWhp1qocakQ07bhME8TMq+KVa5bWoxBbMC66LoN75owK93UDh/TQrKHjabqWamav9A ClNLHAtChMS3+g3mNO7SWuuDp6ktmLQce1POFrKTrZLM93ane7K32OiPAx2Ddd1fna+v NFP7eb+HOkTp7cYzs9KiwL0rFBLYsKcSI+4BRRcZFixpb0aWvSJNAt+A5OD/ZaeSswlD H+xEbvPGriOwbKxVF8+Vz0CELUqLrlYH+BGAhVWnd6thNi9iy8uu8oFjkIW7QEFfXjHc VHbLHa9Zu0ObzMRaaviUDOqgnup9N3awOGSLSnlo/qC26jz6NzzFJtqCtYXyROvq0KhY UnsQ== X-Gm-Message-State: AGi0PuYBZ4zL5pqxnawG9Re7PYbHngs2y4INLTlqrx6zgro2agvI+Zu0 d+ELCXuv+rg7bVWgZnAx3s+h/Vppwhu8Isk8ZPxcvBSXRD4NXQ== X-Google-Smtp-Source: APiQypICXjqhgGQfT5RR2p+dqnq/9hB5Uh6XJ2UEZL0jN6Pd/FZ8UWV035wXTs4/z4vmqmaKrS7u/UQPPrH2e8jQEwY= X-Received: by 2002:a2e:a312:: with SMTP id l18mr7868217lje.229.1585303242321; Fri, 27 Mar 2020 03:00:42 -0700 (PDT) MIME-Version: 1.0 From: Yangze Guo Date: Fri, 27 Mar 2020 18:00:31 +0800 Message-ID: Subject: [Third-party Tool] Flink memory calculator To: user , user-zh@flink.apache.org Content-Type: text/plain; charset="UTF-8" Hi, there. In release-1.10, the memory setup of task managers has changed a lot. I would like to provide here a third-party tool to simulate and get the calculation result of Flink's memory configuration. Although there is already a detailed setup guide[1] and migration guide[2] officially, the calculator could further allow users to: - Verify if there is any conflict in their configuration. The calculator is more lightweight than starting a Flink cluster, especially when running Flink on Yarn/Kubernetes. User could make sure their configuration is correct locally before deploying it to external resource managers. - Get all of the memory configurations before deploying. User may set taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. But they also want to know the total memory consumption of Flink. With this tool, users could get all of the memory configurations they are interested in. If anything is unexpected, they would not need to re-deploy a Flink cluster. The repo link of this tool is https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the BashJavaUtils.jar of Flink and ensures the calculation result is exactly the same as your Flink dist. For more details, please take a look at the README. Any feedback or suggestion is welcomed! [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html Best, Yangze Guo