Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 37A7B18E16 for ; Tue, 23 Feb 2016 10:57:33 +0000 (UTC) Received: (qmail 91396 invoked by uid 500); 23 Feb 2016 10:57:32 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 91307 invoked by uid 500); 23 Feb 2016 10:57:32 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 91296 invoked by uid 99); 23 Feb 2016 10:57:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2016 10:57:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 8CC1D1A426A for ; Tue, 23 Feb 2016 10:57:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.448 X-Spam-Level: * X-Spam-Status: No, score=1.448 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Q56be00NCbGS for ; Tue, 23 Feb 2016 10:57:28 +0000 (UTC) Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 05EC25FAC6 for ; Tue, 23 Feb 2016 10:57:28 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id 9so210584561iom.1 for ; Tue, 23 Feb 2016 02:57:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=YsqOi8u3sI2GknCTR0z5dRGC4Pef3bJ7tpOMZVI1nf4=; b=sIRMZYvVgTjeecGpsRh7rtFkJn0FCGu+MuZFyRhy8KBizUE19xX1i/sp5k96eMyEOB NHz7HtUznQW8TN28NB0VtDIQuj/IuDsW0vzT9FoOaqFCo3eb17vvWJppRD9pht2e4nY+ 4wFh3/J/hNX1PrKGj7or0Tid7l/zzYXh4ceFYvIVvcaDTZDqj+aO6HUSpmHuHM2ZE65b xLsJJv0n+gdyceUvyUkG0EZ7/pdIsK/pucD3WM9ZLimhfMeMj6mD0887mLq659in0Ivo 1LMUqzqf8jnEin1MwyDZsbFmQb52/xUMM+gL/FsRSVQuXyNOIvCFF8npia9eYlyr+kUv DSYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=YsqOi8u3sI2GknCTR0z5dRGC4Pef3bJ7tpOMZVI1nf4=; b=FFV/6DIII708FyQNaz2swse8PZtRyPUNfdj+Fj4Hy1Wqaps4LJnG68w1sWUvfqdM0V pDszHxwkW8PEt33P60Ajgyc8zOgnKfSpWhwnsXqAt4VrDcTecD4oX8yt2QV4r8M5Gprq l3Eo9ijapS/YoAwD2+L1se1LqRPRF20L5T74tVNASJRApYa17Kdy1L2/iajundL/PmAl k5x90q4sypx/8ZQrv42LGp3/9rC3d0qgNzRuX2TrmsPr/GVdodzMvdxxzIosELfyQhlV sStncOyC3ajh8d59mGGLIxmyqOCEqWYdAB7Cq1Bz1jGQheyI6QTgAqLzeIoTguLYPxOE gUaw== X-Gm-Message-State: AG10YOSpsoDuSj0fTneLtgB8F2r4Obxt5uCcBJRHB/a0ahP0JCJNTpvXie9+0UpEI6TapGPURCNl1g4oBqMT9A== MIME-Version: 1.0 X-Received: by 10.107.170.79 with SMTP id t76mr40586768ioe.71.1456225041520; Tue, 23 Feb 2016 02:57:21 -0800 (PST) Received: by 10.107.1.197 with HTTP; Tue, 23 Feb 2016 02:57:21 -0800 (PST) In-Reply-To: References: Date: Tue, 23 Feb 2016 17:57:21 +0700 Message-ID: Subject: Re: Optimal Configuration for Cluster From: Welly Tambunan To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a114157c6501ef8052c6dd051 --001a114157c6501ef8052c6dd051 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Ufuk and Fabian, Is that better to start 48 task manager ( one slot each ) in one machine than having single task manager with 48 slot ? Any trade-off that we should know etc ? Cheers On Tue, Feb 23, 2016 at 3:03 PM, Welly Tambunan wrote: > Hi Ufuk, > > Thanks for the explanation. > > Yes. Our jobs is all streaming job. > > Cheers > > On Tue, Feb 23, 2016 at 2:48 PM, Ufuk Celebi wrote: > >> The new default is equivalent to the previous "streaming mode". The >> community decided to get rid of this distinction, because it was >> confusing to users. >> >> The difference between "streaming mode" and "batch mode" was how >> Flink's managed memory was allocated, either lazily when required >> ('streaming mode") or eagerly on task manager start up ("batch mode"). >> Now it's lazy by default. >> >> This is not something you need to worry about, but if you are mostly >> using the DataSet API where pre allocation has benefits, you can get >> the "batch mode" behaviour by using the following configuration key: >> >> taskmanager.memory.preallocate: true >> >> But you are using the DataStream API anyways, right? >> >> =E2=80=93 Ufuk >> >> >> On Tue, Feb 23, 2016 at 6:36 AM, Welly Tambunan >> wrote: >> > Hi Fabian, >> > >> > Previously when using flink 0.9-0.10 we start the cluster with streami= ng >> > mode or batch mode. I see that this one is gone on Flink 1.00 snapshot >> ? So >> > this one has already taken care of the flink and optimize by runtime > >> > >> > On Mon, Feb 22, 2016 at 5:26 PM, Fabian Hueske >> wrote: >> >> >> >> Hi Welly, >> >> >> >> sorry for the late response. >> >> >> >> The number of network buffers primarily depends on the maximum >> parallelism >> >> of your job. >> >> The given formula assumes a specific cluster configuration (1 task >> manager >> >> per machine, one parallel task per CPU). >> >> The formula can be translated to: >> >> >> >> taskmanager.network.numberOfBuffers: p ^ 2 * t * 4 >> >> >> >> where p is the maximum parallelism of the job and t is the number of >> task >> >> manager. >> >> You can process more than one parallel task per TM if you configure >> more >> >> than one processing slot per machine ( taskmanager.numberOfTaskSlots)= . >> The >> >> TM will divide its memory among all its slots. So it would be possibl= e >> to >> >> start one TM for each machine with 100GB+ memory and 48 slots each. >> >> >> >> We can compute the number of network buffers if you give a few more >> >> details about your setup: >> >> - How many task managers do you start? I assume more than one TM per >> >> machine given that you assign only 4GB of memory out of 128GB to each >> TM. >> >> - What is the maximum parallelism of you program? >> >> - How many processing slots do you configure for each TM? >> >> >> >> In general, pipelined shuffles with a high parallelism require a lot = of >> >> memory. >> >> If you configure batch instead of pipelined transfer, the memory >> >> requirement goes down >> >> (ExecutionConfig.setExecutionMode(ExecutionMode.BATCH)). >> >> >> >> Eventually, we want to merge the network buffer and the managed memor= y >> >> pools. So the "taskmanager.network.numberOfBuffers" configuration whi= ll >> >> hopefully disappear at some point in the future. >> >> >> >> Best, Fabian >> >> >> >> 2016-02-19 9:34 GMT+01:00 Welly Tambunan : >> >>> >> >>> Hi All, >> >>> >> >>> We are trying to running our job in cluster that has this informatio= n >> >>> >> >>> 1. # of machine: 16 >> >>> 2. memory : 128 gb >> >>> 3. # of core : 48 >> >>> >> >>> However when we try to run we have an exception. >> >>> >> >>> "insufficient number of network buffers. 48 required but only 10 >> >>> available. the total number of network buffers is currently set to >> 2048" >> >>> >> >>> After looking at the documentation we set configuration based on doc= s >> >>> >> >>> taskmanager.network.numberOfBuffers: # core ^ 2 * # machine * 4 >> >>> >> >>> However we face another error from JVM >> >>> >> >>> java.io.IOException: Cannot allocate network buffer pool: Could not >> >>> allocate enough memory segments for NetworkBufferPool (required (Mb)= : >> 2304, >> >>> allocated (Mb): 698, missing (Mb): 1606). Cause: Java heap space >> >>> >> >>> We fiddle the taskmanager.heap.mb: 4096 >> >>> >> >>> Finally the cluster is running. >> >>> >> >>> However i'm still not sure about the configuration and fiddling in >> task >> >>> manager heap really fine tune. So my question is >> >>> >> >>> Am i doing it right for numberOfBuffers ? >> >>> How much should we allocate on taskmanager.heap.mb given the >> information >> >>> Any suggestion which configuration we need to set to make it optimal >> for >> >>> the cluster ? >> >>> Is there any chance that this will get automatically resolve by >> >>> memory/network buffer manager ? >> >>> >> >>> Thanks a lot for the help >> >>> >> >>> Cheers >> >>> >> >>> -- >> >>> Welly Tambunan >> >>> Triplelands >> >>> >> >>> http://weltam.wordpress.com >> >>> http://www.triplelands.com >> >> >> >> >> > >> > >> > >> > -- >> > Welly Tambunan >> > Triplelands >> > >> > http://weltam.wordpress.com >> > http://www.triplelands.com >> > > > > -- > Welly Tambunan > Triplelands > > http://weltam.wordpress.com > http://www.triplelands.com > --=20 Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com --001a114157c6501ef8052c6dd051 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Ufuk and Fabian,=C2=A0

Is that better to start 48 task manager ( one slot ea= ch ) in one machine than having single task manager with 48 slot ? Any trad= e-off that we should know etc ?

Cheers

O= n Tue, Feb 23, 2016 at 3:03 PM, Welly Tambunan <if05041@gmail.com><= /span> wrote:
Hi Ufuk,= =C2=A0

Thanks for the explanation.=C2=A0

Yes. Our= jobs is all streaming job.=C2=A0

Cheers
=
On Tue, Feb 23, 2016 at 2:48 PM, Ufuk Celebi <= span dir=3D"ltr"><uc= e@apache.org> wrote:
The ne= w default is equivalent to the previous "streaming mode". The
community decided to get rid of this distinction, because it was
confusing to users.

The difference between "streaming mode" and "batch mode"= ; was how
Flink's managed memory was allocated, either lazily when required
('streaming mode") or eagerly on task manager start up ("batc= h mode").
Now it's lazy by default.

This is not something you need to worry about, but if you are mostly
using the DataSet API where pre allocation has benefits, you can get
the "batch mode" behaviour by using the following configuration k= ey:

taskmanager.memory.preallocate: true

But you are using the DataStream API anyways, right?

=E2=80=93 Ufuk


On Tue, Feb 23, 2016 at 6:36 AM, Welly Tambunan <if05041@gmail.com> wrote:
> Hi Fabian,
>
> Previously when using flink 0.9-0.10 we start the cluster with streami= ng
> mode or batch mode. I see that this one is gone on Flink 1.00 snapshot= ? So
> this one has already taken care of the flink and optimize by runtime &= gt;
>
> On Mon, Feb 22, 2016 at 5:26 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>>
>> Hi Welly,
>>
>> sorry for the late response.
>>
>> The number of network buffers primarily depends on the maximum par= allelism
>> of your job.
>> The given formula assumes a specific cluster configuration (1 task= manager
>> per machine, one parallel task per CPU).
>> The formula can be translated to:
>>
>> taskmanager.network.numberOfBuffers: p ^ 2 * t * 4
>>
>> where p is the maximum parallelism of the job and t is the number = of task
>> manager.
>> You can process more than one parallel task per TM if you configur= e more
>> than one processing slot per machine ( taskmanager.numberOfTaskSlo= ts). The
>> TM will divide its memory among all its slots. So it would be poss= ible to
>> start one TM for each machine with 100GB+ memory and 48 slots each= .
>>
>> We can compute the number of network buffers if you give a few mor= e
>> details about your setup:
>> - How many task managers do you start? I assume more than one TM p= er
>> machine given that you assign only 4GB of memory out of 128GB to e= ach TM.
>> - What is the maximum parallelism of you program?
>> - How many processing slots do you configure for each TM?
>>
>> In general, pipelined shuffles with a high parallelism require a l= ot of
>> memory.
>> If you configure batch instead of pipelined transfer, the memory >> requirement goes down
>> (ExecutionConfig.setExecutionMode(ExecutionMode.BATCH)).
>>
>> Eventually, we want to merge the network buffer and the managed me= mory
>> pools. So the "taskmanager.network.numberOfBuffers" conf= iguration whill
>> hopefully disappear at some point in the future.
>>
>> Best, Fabian
>>
>> 2016-02-19 9:34 GMT+01:00 Welly Tambunan <if05041@gmail.com>:
>>>
>>> Hi All,
>>>
>>> We are trying to running our job in cluster that has this info= rmation
>>>
>>> 1. # of machine: 16
>>> 2. memory : 128 gb
>>> 3. # of core : 48
>>>
>>> However when we try to run we have an exception.
>>>
>>> "insufficient number of network buffers. 48 required but = only 10
>>> available. the total number of network buffers is currently se= t to 2048"
>>>
>>> After looking at the documentation we set configuration based = on docs
>>>
>>> taskmanager.network.numberOfBuffers: # core ^ 2 * # machine * = 4
>>>
>>> However we face another error from JVM
>>>
>>> java.io.IOException: Cannot allocate network buffer pool: Coul= d not
>>> allocate enough memory segments for NetworkBufferPool (require= d (Mb): 2304,
>>> allocated (Mb): 698, missing (Mb): 1606). Cause: Java heap spa= ce
>>>
>>> We fiddle the taskmanager.heap.mb: 4096
>>>
>>> Finally the cluster is running.
>>>
>>> However i'm still not sure about the configuration and fid= dling in task
>>> manager heap really fine tune. So my question is
>>>
>>> Am i doing it right for numberOfBuffers ?
>>> How much should we allocate on taskmanager.heap.mb given the i= nformation
>>> Any suggestion which configuration we need to set to make it o= ptimal for
>>> the cluster ?
>>> Is there any chance that this will get automatically resolve b= y
>>> memory/network buffer manager ?
>>>
>>> Thanks a lot for the help
>>>
>>> Cheers
>>>
>>> --
>>> Welly Tambunan
>>> Triplelands
>>>
>>> http://weltam.wordpress.com
>>> http://www.triplelands.com
>>
>>
>
>
>
> --
> Welly Tambunan
> Triplelands
>
> http://weltam.wordpress.com
> http://www.triplelands.com



--
=



--
=
Welly Tambunan
Triplelands=C2=A0

<= a href=3D"http://weltam.wordpress.com" target=3D"_blank">http://weltam.word= press.com
--001a114157c6501ef8052c6dd051--