From user-zh-return-1518-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Nov 20 07:08:47 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3D89B1804BB for ; Wed, 20 Nov 2019 08:08:47 +0100 (CET) Received: (qmail 40366 invoked by uid 500); 20 Nov 2019 07:08:37 -0000 Mailing-List: contact user-zh-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-zh@flink.apache.org Delivered-To: mailing list user-zh@flink.apache.org Received: (qmail 39964 invoked by uid 99); 20 Nov 2019 07:08:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Nov 2019 07:08:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A4E61180EED; Wed, 20 Nov 2019 07:08:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id StLvkmUC2Kxd; Wed, 20 Nov 2019 07:08:33 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.41; helo=mail-wm1-f41.google.com; envelope-from=liujiangangpeng@gmail.com; receiver= Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 1D20DBC509; Wed, 20 Nov 2019 07:08:33 +0000 (UTC) Received: by mail-wm1-f41.google.com with SMTP id l17so5807919wmh.0; Tue, 19 Nov 2019 23:08:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=pfDpp6KilbArnGyh7xT+YjsRWAWdUMPZnXwfwMEoLYg=; b=ns4q0eclEtoDh2sn1VwhOaN8N54GJVjY79QeVm2KR3epLvwaRm2fCCXknvkEB5r7qe JqdzKtBFvyGOQPguTnD0UDDPWW8gVfLEnpt7YmAEewMH1+0ZNfzKiApNNYs073uCmW+O mbdpFxbOV4BDFMBTQmcVveM3t7W4hndBuNcAokjGe5ZBDTfOUrpaN2fxcLh0Umdb6383 Tqiost92L5lJBdPzaUTDjb2wSUzLpyBa9Lsw5CRgkNu94ZOTDCS0y6p1e74s1pQA6vBu RjpWiZAyIFbho6Decbqew7jHDPHOT/0pL0wOBXXIPWIx4kFm2spBu/e+M2slfCe5EOXm W6MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=pfDpp6KilbArnGyh7xT+YjsRWAWdUMPZnXwfwMEoLYg=; b=Fl3q4CPcEyzVpqNFEt0AZQ9zyDFdLyXAUurrK/xPT04zsME52aPR86tvHgIAXJt3p7 QwPLPNQZNDRnBdVb3iKxy9w2ik9fkcqRJw4iGVehyYvnC7qFrsEQrjFq2DUoGNE4mNUk 1gj7wFN0M5UpAxk/QXNXuosQzaC2qOTihiA7kT2Rm0c90D4Jw4E5vIAl6Ow/X6eSXBXj TF/RBopOIxp55f0dKUTxEngGXqEvc3tTerzG474VGX7j1S9mY9PqWbTFgZ0fkYe64g0t VlYTVVj7dW7BCYpTAa4j470qtociA5tLXm9Ph24aQCQ0a0Q3ghwTEke4p+0KyYoUCQ6i g15Q== X-Gm-Message-State: APjAAAXTW7/bnS5DRpfTBoqIQHhr6izWdV/74gUbcP7n2U3zBulpk0y2 3s6sop1S8bPpXDnIBLnnSYmggMdST6nobHrS0Zs7PQ2l X-Google-Smtp-Source: APXvYqw/Mrj4oJm1Q44CkRaHkVZIAoNRzlXlHm5NAYWWZAaIoA5EaM0q8xa4oG/J70SOK2Wz+/qdnlJQmEbJWgXSjiU= X-Received: by 2002:a1c:9c54:: with SMTP id f81mr1358926wme.89.1574233711767; Tue, 19 Nov 2019 23:08:31 -0800 (PST) MIME-Version: 1.0 From: =?UTF-8?B?5YiY5bu65Yia?= Date: Wed, 20 Nov 2019 15:08:20 +0800 Message-ID: Subject: How to estimate the memory size of flink state To: dev@flink.apache.org, user-zh@flink.apache.org, user@flink.apache.org Content-Type: multipart/alternative; boundary="0000000000002ef7070597c1d98d" --0000000000002ef7070597c1d98d Content-Type: text/plain; charset="UTF-8" We are using flink 1.6.2. For filesystem backend, we want to monitor the state size in memory. Once the state size becomes bigger, we can get noticed and take measures such as rescaling the job, or the job may fail because of the memory. We have tried to get the memory usage for the jvm, like gc throughput. For our case, state can vary greatly at the peak. So maybe I can refer to the state memory size. I checked the metrics and code, but didn't find any information about the state memory size. I can get the checkpoint size, but they are serialized result that can not reflect the running state in memory. Can anyone give me some suggestions? Thank you very much. --0000000000002ef7070597c1d98d--