From user-return-33790-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Mar 30 04:03:09 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1829D180644 for ; Mon, 30 Mar 2020 06:03:08 +0200 (CEST) Received: (qmail 67747 invoked by uid 500); 30 Mar 2020 04:03:05 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 67725 invoked by uid 99); 30 Mar 2020 04:03:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Mar 2020 04:03:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 836D21A00DD; Mon, 30 Mar 2020 04:03:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.2 X-Spam-Level: X-Spam-Status: No, score=-0.2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id UIp7UGmSGw8W; Mon, 30 Mar 2020 04:03:03 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::244; helo=mail-lj1-x244.google.com; envelope-from=karmagyz@gmail.com; receiver= Received: from mail-lj1-x244.google.com (mail-lj1-x244.google.com [IPv6:2a00:1450:4864:20::244]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id C30A57DC60; Mon, 30 Mar 2020 04:03:02 +0000 (UTC) Received: by mail-lj1-x244.google.com with SMTP id k21so16592648ljh.2; Sun, 29 Mar 2020 21:03:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=FRk1DR4nXyHPbV1Am+Hso6mQoL5AQB8dDOE8CEnvSeM=; b=T6tycIHDKIkX1BV0MzDIARDUciSgq4TknOJtvn9Cz/JK7n0doOkZuIqWhjsJntveOa yu3x8ZzSAhnJhHz4WnQORp6bdllmhKY53HFexY92VFmlMkXvUX5ROa15woc+/oMdJBn/ TJL1g8TT1FiPjU5b471ueEu64mp8OgU9hLZfna2qMuiRI9XP7IQldme8POBgqfhUpJ4x crA1TXl+xwWxjYdHYnimkpwoQXYMtLpu8zqyXHQiPdQgqtiGsRJeQ7yAKbFrcltnRgiS zkblS1B57LbswqwYGAfNLce5+jXYbCR5ELMDjj/W9h/nEYYI5rCa8sWQ25wF2kiACJxb PUHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=FRk1DR4nXyHPbV1Am+Hso6mQoL5AQB8dDOE8CEnvSeM=; b=DX4RK20KE5XonrONlQg0mWv9jflbmP3ibZW6TBZleFMb4v7QhtuGwq1Wy4wZwxdLhU Am3E20PiuYWY0DN2S82gOHBYddPKO0PgCNgWAlKm4s/1xN4N2/5rm0NkeTj64AP9Wltl tyXEF3cxkH44ZAOZCvumCxkLeNSqTtLh2pm4psEKbkv9IJ4bOSdLo6KxuIylAY6/TIYE Xxc1pexZ5vJjq5ecWClZTz/oXq3rsurf1h5uV2u91pNMTIWf5KO4O9RhuIGXDVyKgqRi m/Bu14/I8mTn03rYYuCfoh1eqD6XOaTgH9eXAxxPvmRduR4X54LWgwSjlUmg/FtxUqCv F3/Q== X-Gm-Message-State: AGi0PuZGJRY6XhS4U4TGniy2znsqqMvDPdOR7zPtJmQGJkU8ho0iiuXq d/gcHYu9W+hTRH4+cz9kR4ZDH48vMCO9Rg0jf70= X-Google-Smtp-Source: APiQypJVPdjlnOHQlHJG1Y28tUVAemq/4wxyNz7EzIPwu7XdthqUfVpBnWJMnWxqxKKcwQrNbwIZEeaDxu/EaL312+g= X-Received: by 2002:a2e:99c8:: with SMTP id l8mr5623169ljj.48.1585540982240; Sun, 29 Mar 2020 21:03:02 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Yangze Guo Date: Mon, 30 Mar 2020 12:02:51 +0800 Message-ID: Subject: Re: [Third-party Tool] Flink memory calculator To: Jeff Zhang Cc: Yun Tang , user , "user-zh@flink.apache.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for your feedbacks, @Xintong and @Jeff. @Jeff I think it would always be good to leverage exist logic in Flink, such as JobListener. However, this calculator does not only target to check the conflict, it also targets to provide the calculating result to user before the job is actually deployed in case there is any unexpected configuration. It's a good point that we need to parse the dynamic configs. I prefer to parse the dynamic configs and cli commands in bash instead of adding hook in JobListener. Best, Yangze Guo On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > > Hi Yangze, > > Does this tool just parse the configuration in flink-conf.yaml ? Maybe i= t could be done in JobListener [1] (we should enhance it via adding hook be= fore job submission), so that it could all the cases (e.g. parameters comin= g from command line) > > [1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/= org/apache/flink/core/execution/JobListener.java#L35 > > > Yangze Guo =E4=BA=8E2020=E5=B9=B43=E6=9C=8830=E6=97= =A5=E5=91=A8=E4=B8=80 =E4=B8=8A=E5=8D=889:40=E5=86=99=E9=81=93=EF=BC=9A >> >> Hi, Yun, >> >> I'm sorry that it currently could not handle it. But I think it is a >> really good idea and that feature would be added to the next version. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> > >> > Very interesting and convenient tool, just a quick question: could thi= s tool also handle deployment cluster commands like "-tm" mixed with config= uration in `flink-conf.yaml` ? >> > >> > Best >> > Yun Tang >> > ________________________________ >> > From: Yangze Guo >> > Sent: Friday, March 27, 2020 18:00 >> > To: user ; user-zh@flink.apache.org >> > Subject: [Third-party Tool] Flink memory calculator >> > >> > Hi, there. >> > >> > In release-1.10, the memory setup of task managers has changed a lot. >> > I would like to provide here a third-party tool to simulate and get >> > the calculation result of Flink's memory configuration. >> > >> > Although there is already a detailed setup guide[1] and migration >> > guide[2] officially, the calculator could further allow users to: >> > - Verify if there is any conflict in their configuration. The >> > calculator is more lightweight than starting a Flink cluster, >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> > their configuration is correct locally before deploying it to external >> > resource managers. >> > - Get all of the memory configurations before deploying. User may set >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> > But they also want to know the total memory consumption of Flink. With >> > this tool, users could get all of the memory configurations they are >> > interested in. If anything is unexpected, they would not need to >> > re-deploy a Flink cluster. >> > >> > The repo link of this tool is >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> > exactly the same as your Flink dist. For more details, please take a >> > look at the README. >> > >> > Any feedback or suggestion is welcomed! >> > >> > [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/= mem_setup.html >> > [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/= mem_migration.html >> > >> > Best, >> > Yangze Guo > > > > -- > Best Regards > > Jeff Zhang