From user-return-31642-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Dec 25 09:16:24 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D3287180629 for ; Wed, 25 Dec 2019 10:16:23 +0100 (CET) Received: (qmail 7585 invoked by uid 500); 25 Dec 2019 09:16:22 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 7575 invoked by uid 99); 25 Dec 2019 09:16:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Dec 2019 09:16:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 760ACC0827 for ; Wed, 25 Dec 2019 09:16:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id gJTmQG4YUvJa for ; Wed, 25 Dec 2019 09:16:20 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::243; helo=mail-lj1-x243.google.com; envelope-from=iloahz@gmail.com; receiver= Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id A979C7E103 for ; Wed, 25 Dec 2019 09:16:19 +0000 (UTC) Received: by mail-lj1-x243.google.com with SMTP id u1so22484451ljk.7 for ; Wed, 25 Dec 2019 01:16:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6p/qJwtEsju+xGZAvavSQsA1ypLz/JHJmPKomPR3UDc=; b=XQYH0gAevz60Cc8qnsNwDXtaPa0eeGBKvRoiiy9SPzXdQPRID6M+1IwNiZcok0wMgF pazm0xmsLkAk8NDZi4rJ0Uf+5iTpKf5RjMpVe2iSIj/KM1ZhDjlyuOVUlDmox1kteqJx Yp6qpRfsJzWSw11VoIS5KTmKqa7b3pEqTmREO5Df3MDnc/SGyfQ3kugAyCV8CLeWYLDX 1vuQqjlecy9kf5hxD5yv2T3puRQsM0d4JusSi2o7IdawX+BsqOL9+SD8fZcGd6gRIx1z DJVFlFTent04WjaNzAnnzOHzaTH7azD3koO84ouYRl621yynvrkt3KPdVBlseG05V5HX Id9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6p/qJwtEsju+xGZAvavSQsA1ypLz/JHJmPKomPR3UDc=; b=KWoM7vOuVOxOQZ24vtw0zXLX2cHLqgUbxr1MzeFqDFhDiwHGIHVqE+P6QReUvCeiae Xb7S3ZUdvItSwHjggosxcHpO9AzWXz5N5fz/N3S9ORhSTK64Qcgem2kFaL3G8mvd83kp 47kXj55mT9m3tDtt2urREkCi/XKZRbCg7LVPoNMUmYriM3nwr5FOGKqujV9h/N9HJSTB 01mkzFSpotPaoVPyHfihEHZ5pzvrl5o3rcbRau0jgNDKnOfEf3RK1F5ewBjUzzLtYF5I 7Zio+U4cxugxL6wxeWbRN/GpfUGkItPGxJM/gPD/VGmwYJ3rf4h5Drv2/hBJ5i9+Di6c mr0Q== X-Gm-Message-State: APjAAAUpXHvy3D2Y9Aim+uMlJP9Xt/+8WGgxaYqC2TQlHRxH/6VjAEFa JjwB0JKeRhr+PmGUhNV9WGHvuT1lw9J7xaz7BUxZvpPHjXM= X-Google-Smtp-Source: APXvYqzKIuVLV0d5cb4dl0ibzQlq61SGby1aVbOn+4ZrTrTN1k64fCDNheWmMJbyNAHHLG85EqUFwH7qkQAR3DiX/JM= X-Received: by 2002:a2e:9050:: with SMTP id n16mr12245357ljg.49.1577265379014; Wed, 25 Dec 2019 01:16:19 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Li Zhao Date: Wed, 25 Dec 2019 17:16:07 +0800 Message-ID: Subject: Re: question: jvm heap size per task? To: Xintong Song Cc: user Content-Type: multipart/alternative; boundary="000000000000a1feaf059a83b6c3" --000000000000a1feaf059a83b6c3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Understood, thank you for the quick response! Thanks, Li Xintong Song =E4=BA=8E2019=E5=B9=B412=E6=9C=8825=E6= =97=A5=E5=91=A8=E4=B8=89 =E4=B8=8B=E5=8D=885:05=E5=86=99=E9=81=93=EF=BC=9A > Hi Li, > > It is true that currently all the task managers have the same heap size, > and it's fixed ever since started. Unfortunately, your needs cannot be > satisfied at the moment. > > Heap size of task managers cannot be changed once started, because flink > task managers run in JVMs and JVM does not support resizing after started= . > > However, there is an ongoing approach towards your needs. The community i= s > working on fine-grained resource management, which in general allows > specify per task/operator resource requirements and allocate task manager > resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for > more details. Another related effort is pluggable slot manager [3], which > allows having pluggable resource scheduling strategies such as launch tas= k > managers with customized resources according to the tasks' requirements. > > Thank you~ > > Xintong Song > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained= +Operator+Resource+Management > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot= +Allocation > [3] https://issues.apache.org/jira/browse/FLINK-14106 > > On Wed, Dec 25, 2019 at 4:18 PM Li Zhao wrote: > >> Hi, >> >> Greetings, hope this is the proper place to ask questions, apologize if >> not. >> >> We have a shared flink cluster running with docker, want to set differen= t >> heap size per task(some tasks require larger heap size, while most tasks >> only need a little), is it feasible? >> >> I've gone through [1], [2] and [3], my current understanding is all task >> managers have the same heap size which is set by `taskmanager.heap.size` >> and is fixed ever since the task manager is started, and all tasks runni= ng >> in that task manager will share that heap size. >> >> Am I understanding it correctly? And any approaches to our needs? >> >> Thanks in advance! >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html >> [2]: >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/run= time.html >> [3]: >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Mem= ory+Configuration+for+TaskExecutors >> >> Thanks, >> Li >> > --000000000000a1feaf059a83b6c3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Understood, thank you for the quick=C2=A0response!
Thanks,
Li

=
Xintong Song <tonysong820@gmail.com> =E4=BA=8E2019=E5=B9=B412= =E6=9C=8825=E6=97=A5=E5=91=A8=E4=B8=89 =E4=B8=8B=E5=8D=885:05=E5=86=99=E9= =81=93=EF=BC=9A
=
Hi Li,

It is true that currently all th= e task managers have the same heap size, and it's fixed ever since star= ted. Unfortunately, your needs cannot be satisfied at the moment.

Heap size of task managers cannot be changed once started, = because flink task managers run in JVMs and JVM does not support resizing a= fter started.

However, there is an ongoing approac= h towards your needs. The community is working on fine-grained resource man= agement, which in general allows specify per task/operator resource require= ments and allocate task manager resources accordingly. You can refers to FL= IP-53 [1] and FLIP-56 [2] for more details. Another related effort is plugg= able slot manager [3], which allows having pluggable resource scheduling st= rategies such as launch task managers with customized resources according= =C2=A0to the tasks' requirements.

<= div dir=3D"ltr">

Thank you~

Xinto= ng Song



On Wed, Dec 25, 2019 at 4= :18 PM Li Zhao <il= oahz@gmail.com> wrote:
Hi,

Greetings, hope this = is the proper place to ask questions, apologize if not.

We have a shared flink cluster running with docker, want to set diffe= rent heap size per task(some tasks require larger heap size, while most tas= ks only need a little), is it feasible?

I've g= one through [1], [2] and [3], my current understanding is all task managers= have the same heap size which is set by `taskmanager.heap.size` and is fix= ed ever since the task manager is started, and all tasks running in that ta= sk manager will share that heap size.

Am I underst= anding it correctly? And any approaches to our needs?

<= div>Thanks in advance!


Thanks,
Li
--000000000000a1feaf059a83b6c3--