From user-zh-return-1183-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Sep 23 06:39:27 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id CFF4C180651 for ; Mon, 23 Sep 2019 08:39:26 +0200 (CEST) Received: (qmail 24730 invoked by uid 500); 23 Sep 2019 06:39:26 -0000 Mailing-List: contact user-zh-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-zh@flink.apache.org Delivered-To: mailing list user-zh@flink.apache.org Received: (qmail 24717 invoked by uid 99); 23 Sep 2019 06:39:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Sep 2019 06:39:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 44F261A42B4 for ; Mon, 23 Sep 2019 06:39:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.051 X-Spam-Level: ** X-Spam-Status: No, score=2.051 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id C3knY5PHTgaN for ; Mon, 23 Sep 2019 06:39:23 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::d43; helo=mail-io1-xd43.google.com; envelope-from=mmyy1110@gmail.com; receiver= Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 5D8C07DD69 for ; Mon, 23 Sep 2019 06:39:22 +0000 (UTC) Received: by mail-io1-xd43.google.com with SMTP id a1so30753100ioc.6 for ; Sun, 22 Sep 2019 23:39:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=OTNB8ebF+7Ombu1tWzsfTD9ETo6vnmVfbqbgns/Hy3c=; b=lf82P1/RoGnBPvf5CEuW+Jr3SGFytsVSpKluudZSBYgpF95fDRgeX/R6OUbjxq4Bh6 wO6JxcIq0gPI6oevqA3hg+TIC4UL+FZoOjeTwWCUyIgyz/999g8rAFfCArg2f6HJ2g6p EY1DLLzw983kkjuZeEX722jy2TU6kKvN8f/Rd3yYTZ2pSxb0gghncrnKXs4ykQ2zE9gp pzXgtG+FzBKb70EXngEYh3mEPJjVbdrdjS10gA0enoe06QWhw4jMhOKe7x5jxmBX/oWf HjApf4EdM5yl7kgtV2K+kPbk2Ny/GB+b5SYb0LAwC4REMa+QbP45jJFWnz0blTuYE6kQ AlJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=OTNB8ebF+7Ombu1tWzsfTD9ETo6vnmVfbqbgns/Hy3c=; b=Mav8HpKN1/udqbm91JT+JRaJmVlih0ych7jCDwQrp3ipEmUQzfPz3bd6aofBnItev3 0p8tatXGXZkhfVkeyQOrsuXV3qBROHPYLKdsy15iYHX4MZCFcoXM0lf/XYZkyTR6TaQG RI5vqtydA6QkJ9LajtiP9SxkN7gXTxfCwe8kU+ygYR3witY1OL/FVLLzPr/Skn8h3JOT TXlJoHduuXMtSuF1RUxRRJu7e1KvKn6NsWwmUDsiEY0biXmcYGymb6tt2Wn37vcVgEhY USVzXbc6XdVInQe90GBH4Abbrr5YhlKk1cDSe04iFSo7X+Rz8HU4SgElIE+bCBCepDeN gszg== X-Gm-Message-State: APjAAAXWejWVGyoybg68j7aZS/jt04aWwbLzfleOrFYyx7S3IPOa1+f5 LGd9dBeWgLgwu+HP2fP70rrz79QPT28sVtMw0BMoCYlS X-Google-Smtp-Source: APXvYqx2UFTI7XN3fZkyRjUn9R+sditARx8k0IcGvH2jSPnFCnpWWA74UTygtONzVcNADTr9QTiErOS3pKJorECfvkQ= X-Received: by 2002:a02:6616:: with SMTP id k22mr33925927jac.129.1569220760866; Sun, 22 Sep 2019 23:39:20 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Biao Liu Date: Mon, 23 Sep 2019 14:39:10 +0800 Message-ID: Subject: =?UTF-8?B?UmU6IEZsaW5r5aSnc3RhdGXor7vlj5bno4Hnm5jvvIzno4Hnm5hJT+aJk+a7oe+8jOS7uw==?= =?UTF-8?B?5Yqh55u45LqS5b2x5ZON55qE6Zeu6aKY5o6i6K6o?= To: user-zh Content-Type: multipart/alternative; boundary="00000000000006a49e059332ae4c" --00000000000006a49e059332ae4c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, IO =E9=87=8F=E8=BF=99=E4=B9=88=E5=A4=A7=E7=AC=A6=E5=90=88=E9=A2=84=E6=9C=9F= =E5=90=97=EF=BC=9F=E8=80=8C=E4=B8=94=E6=98=AF=E8=AF=BB=E7=A1=AC=E7=9B=98=E6= =89=93=E6=BB=A1=E3=80=82 =E6=9C=89=E6=B2=A1=E6=9C=89=E5=B0=9D=E8=AF=95=E8=BF=87=E8=B0=83=E4=BC=98=EF= =BC=9F 1. =E4=B8=9A=E5=8A=A1=E6=96=B9=E9=9D=A2=E7=9A=84=E8=B0=83=E4=BC=98=EF=BC=8C= =E4=BE=8B=E5=A6=82=E5=AF=B9 state =E7=9A=84=E4=BD=BF=E7=94=A8=E6=98=AF=E5= =90=A6=E5=90=88=E7=90=86 2. =E7=B3=BB=E7=BB=9F=E5=B1=82=E9=9D=A2=E7=9A=84=E8=B0=83=E4=BC=98=EF=BC=8C= =E4=BE=8B=E5=A6=82 incremental checkpoint [1] 1. https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/stat= e/checkpointing.html#state-backend-incremental Thanks, Biao /'b=C9=AA.a=CA=8A/ On Tue, 10 Sep 2019 at 14:39, Wesley Peng wrote: > > > on 2019/9/10 13:47, =E8=92=8B=E6=B6=9B=E6=B6=9B wrote: > > =E5=B0=9D=E8=AF=95=E6=89=8B=E6=AE=B5=EF=BC=9A > > > > 1. =E6=89=8B=E5=8A=A8=E8=BF=81=E7=A7=BBIO=E6=AF=94=E8=BE=83=E9=AB=98=E7= =9A=84=E4=BB=BB=E5=8A=A1=E5=88=B0=E5=85=B6=E4=BB=96=E6=9C=BA=E5=99=A8=EF=BC= =8C=E4=BD=86=E6=98=AFyarn=E4=BB=BB=E5=8A=A1=E6=8F=90=E4=BA=A4=E6=AF=94=E8= =BE=83=E9=9A=8F=E6=9C=BA=EF=BC=8C=E5=8F=AA=E8=83=BD=E5=81=B6=E5=B0=94=E4=B8= =BA=E4=B9=8B > > > > 2. =E7=9B=AE=E5=89=8D=E6=B2=A1=E6=9C=89SSD=EF=BC=8C=E5=8F=AA=E8=83=BD= =E7=94=A8=E6=99=AE=E9=80=9ASTATA=E7=9B=98=EF=BC=8C=E7=9B=AE=E5=89=8D=E5=8A= =A0=E4=BA=86=E4=B8=A4=E5=9D=97=E7=9B=98=E6=8F=90=E7=A4=BA=E7=A3=81=E7=9B=98= IO=E8=83=BD=E5=8A=9B=EF=BC=8C=E4=BD=86=E6=98=AF=E5=8D=95=E7=9B=98=E5=AF=B9= =E5=8D=95=E4=BB=BB=E5=8A=A1=E7=9A=84=E7=A3=81=E7=9B=98IO=E7=93=B6=E9=A2=88= =E8=BF=98=E5=9C=A8 > > > > =E8=BF=98=E6=9C=89=E5=93=AA=E4=BA=9B=E7=AD=96=E7=95=A5=E5=8F=AF=E4=BB= =A5=E8=A7=A3=E5=86=B3=E6=88=96=E8=80=85=E7=BC=93=E8=A7=A3=E4=B9=88=EF=BC=9F > > It seems the tricks to improve RocksDB's throughput might be helpfu. > > With writes and reads accessing mostly the recent data, our goal is to > let them stay in memory as much as possible without using up all the > memory on the server. The following parameters are worth tuning: > > Block cache size: When uncompressed blocks are read from SSTables, they > are cached in memory. The amount of data that can be stored before > eviction policies apply is determined by the block cache size. The > bigger the better. > > Write buffer size: How big can Memtable get before it is frozen. > Generally, the bigger the better. The tradeoff is that big write buffer > takes more memory and longer to flush to disk and to recover. > > Write buffer number: How many Memtables to keep before flushing to > SSTable. Generally, the bigger the better. Similarly, the tradeoff is > that too many write buffers take up more memory and longer to flush to > disk. > > Minimum write buffers to merge: If most recently written keys are > frequently changed, it is better to only flush the latest version to > SSTable. This parameter controls how many Memtables it will try to merge > before flushing to SSTable. It should be less than the write buffer > number. A suggested value is 2. If the number is too big, it takes > longer to merge buffers and there is less chance of duplicate keys in > that many buffers. > > The list above is far from being exhaustive, but tuning them correctly > can have a big impact on performance. Please refer to RocksDB=E2=80=99s T= uning > Guide for more details on these parameters. Figuring out the optimal > combination of values for all of them is an art in itself. > > please ref: https://klaviyo.tech/flinkperf-c7bd28acc67 > > regards. > --00000000000006a49e059332ae4c--