Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A34AA18BDC for ; Wed, 14 Oct 2015 17:15:11 +0000 (UTC) Received: (qmail 84244 invoked by uid 500); 14 Oct 2015 17:15:08 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 84164 invoked by uid 500); 14 Oct 2015 17:15:08 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 84154 invoked by uid 99); 14 Oct 2015 17:15:08 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2015 17:15:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 037001A2314 for ; Wed, 14 Oct 2015 17:15:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.981 X-Spam-Level: *** X-Spam-Status: No, score=3.981 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id fyuXNS7cpOwA for ; Wed, 14 Oct 2015 17:15:00 +0000 (UTC) Received: from mail-yk0-f179.google.com (mail-yk0-f179.google.com [209.85.160.179]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id BD3B920C18 for ; Wed, 14 Oct 2015 17:14:59 +0000 (UTC) Received: by ykey125 with SMTP id y125so53730860yke.3 for ; Wed, 14 Oct 2015 10:14:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=IIs7ejfz1WenLzlt3175XW4EnbwP5cdYCvIUlyBraIs=; b=bc6XVb7IKTGMN7TUZ2NrCHWUCxGaYDVv/HpgoN9Gt2b0fJl26mZzrdanqmf5n+zvq0 3iVsD+Tm9+AgCdbsHD5McdWwcbx/F6o42Nx6TpsHdprRaYUVLN06SE23KOEowF9tZoeD hzgpBWAMgExLE73EQnP2lej4p2c75KRcdBoOPofZ57tjYjnW8xyRUxGIuugWx7QREisv N0AqzSThc9NlEEmxG1ZwciF/V4lV6puBAseC06IiS/0FmqPzaJr4UXWq3YJ0aMgwBjjT VfB0EyTgntNZt5cw0OY/xvrCJEqQV1LRtrU0c2gPisA846wHx0RMLsLpFIZMk+kn7/8l QPFg== X-Gm-Message-State: ALoCoQkG5ZUM2Kw13NJqiWE49a/0LPogUs5hHllDy2QjB3gAHwnisWcOZ3JB2bW70JWOXiQGJ177 MIME-Version: 1.0 X-Received: by 10.13.212.208 with SMTP id w199mr3152802ywd.103.1444842892891; Wed, 14 Oct 2015 10:14:52 -0700 (PDT) Received: by 10.129.84.6 with HTTP; Wed, 14 Oct 2015 10:14:52 -0700 (PDT) X-Originating-IP: [37.227.238.82] Received: by 10.129.84.6 with HTTP; Wed, 14 Oct 2015 10:14:52 -0700 (PDT) In-Reply-To: References: Date: Wed, 14 Oct 2015 19:14:52 +0200 Message-ID: Subject: Re: ExecutionEnvironment setConfiguration API From: Flavio Pompermaier To: user Content-Type: multipart/alternative; boundary=001a114fba5e634e99052213b3e7 --001a114fba5e634e99052213b3e7 Content-Type: text/plain; charset=UTF-8 of course,I tried to configure the task slot during a debug test and I forgot to remove it.. Just for curiosity, is there any good reason why you've changed the default parallellelism that way?and moreover, is it the only unexpected changed behaviour wrt the previous API version? On 14 Oct 2015 18:35, "Stephan Ewen" wrote: > Hi Flavio! > > ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment() > by default picks up the number of cores as the parallelism, while the > manual environments do not do that. > You can still set it manually set the parallelism > "env.setParallelism(Runtime.getRuntime().availableProcessors());" > > I would not configure the slots for the local execution, they should be > automatically configured based on the max parallelism. > > Greetings, > Stephan > > > On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier > wrote: > >> Hi Fabian and Stephan, back to work :) >> >> I finally managed to find the problem of the parallelism encountered by >> my colleague! >> Basically that was introduced by this API change. Before I was using >> env.setConfiguration() to merge the default params with some custom ones. >> Now, after the API change I was using the following code: >> >> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); >> if (env instanceof LocalEnvironment) { >> Configuration c = new Configuration(); >> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR); >> >> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR); >> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2); >> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4); >> env = ExecutionEnvironment.createLocalEnvironment(c); >> } >> >> However, the first env and the reassigned one doesn't behave in the same >> manner. >> If I don't reassign env I have parallelism=8, otherwise it's 1 :( >> Am I using the wrong APIs or the execution environment doesn't allow now >> to configure such parameters anymore? >> >> Thanks in advance, >> Flavio >> >> >> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier > > wrote: >> >>> That makes sense: what can be configured should be differentiated >>> between local and remote envs (obviously this is a minor issue/improvement) >>> >>> Thanks again, >>> Flavio >>> >>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen wrote: >>> >>>> We can think about that, but I think it may be quite confusing. The >>>> configurations actually mean something different for local and remote >>>> environments: >>>> >>>> - For the local environment, the config basically describes the >>>> entire Flink cluster setup (for the local execution cluster in the >>>> background) >>>> - For the remote environment, the config describes the parameters for >>>> the client that connects to the cluster (akka paramters, optimizer >>>> parameters, ...), but not parameters of the cluster itself (like >>>> taskmanager slots and memory). >>>> >>>> Greetings, >>>> Stephan >>>> >>>> >>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier < >>>> pompermaier@okkam.it> wrote: >>>> >>>>> However it could be a good idea to overload also >>>>> the getExecutionEnvironment() to be able to pass a custom >>>>> configuration..what do you think? >>>>> Otherwise I have to know a priori if I'm working in a local deployment >>>>> or in a remote one, or check if getExecutionEnvironment() returned an >>>>> instance of LocalEnvironment/RemoteEnvironment.. >>>>> >>>>> >>>>> >>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier < >>>>> pompermaier@okkam.it> wrote: >>>>> >>>>>> Yes Stephan! >>>>>> I usually work with the master version, at least in development ;) >>>>>> Thanks for the quick support! >>>>>> >>>>>> Best, >>>>>> Flavio >>>>>> >>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen >>>>>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> Are you on the SNAPSHOT master version? >>>>>>> >>>>>>> You can pass the configuration to the constructor of the execution >>>>>>> environment, or create one via >>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via >>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles); >>>>>>> >>>>>>> The change of the signature was part of an API cleanup for the next >>>>>>> release. Sorry for the inconvenience... >>>>>>> >>>>>>> Stephan >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier < >>>>>>> pompermaier@okkam.it> wrote: >>>>>>> >>>>>>>> Hi to all, >>>>>>>> >>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment >>>>>>>> doesn't have setConfiguration() anymore..how can I set the following >>>>>>>> parameters in my unit tests? >>>>>>>> >>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY >>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY >>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY >>>>>>>> >>>>>>>> Best, >>>>>>>> Flavio >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --001a114fba5e634e99052213b3e7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

of course,I tried to configure the task slot during a debug = test and I forgot to remove it..
Just for curiosity, is there any good reason why you've changed the def= ault parallellelism that way?and moreover, is it the only unexpected change= d behaviour wrt the previous API version?

On 14 Oct 2015 18:35, "Stephan Ewen" &= lt;sewen@apache.org> wrote:
Hi Flav= io!

ExecutionEnvironmen= t env =3D ExecutionEnvironment.getE= xecutionEnvironment() by default picks up the number of cores as the parall= elism, while the manual environments do not do that.
<= span style=3D"font-size:12.8px">You can still set it manually set the paral= lelism "env.setParallelism(Runtime.getRuntime().availableProcessors())= ;"

I would not configure the slots for= the local execution, they should be automatically configured based on the = max parallelism.

Greetings,
Stephan


On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier = <pompermaier@o= kkam.it> wrote:
Hi Fabian and Stephan, back to work :)

I finally managed= to find the problem of the parallelism encountered by my colleague!
<= /div>
Basically that was introduced by this API change. Before I was us= ing env.setConfiguration() to merge the default params with some custom one= s.
Now, after the API change I was using the following code:

ExecutionEnvironment env =3D ExecutionEnvironment= .getExecutionEnvironment();
if (env instanceof LocalEnvironment) = {
Configuration c = =3D new Configuration();
c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_D= IR);
c.setString(Con= figConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
c.setLong(ConfigConstants.TASK_MAN= AGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS,= 4);
env =3D Executi= onEnvironment.createLocalEnvironment(c);
}

However, the first en= v and the reassigned one doesn't behave in the same manner.
If I don't reassign env I have parallelism=3D8, ot= herwise it's 1 :(
Am I using the wrong = APIs or the execution environment doesn't allow now to configure such p= arameters anymore?

Thanks in advance,
Flavio

=
On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pomp= ermaier <pompermaier@okkam.it> wrote:
That makes sense: what can be configured should be differ= entiated between local and remote envs (obviously this is a minor issue/imp= rovement)

Thanks again,
Flavio
=

On Tue, Oct 6, 20= 15 at 11:25 AM, Stephan Ewen <sewen@apache.org> wrote:
We can think about that, but I think it may = be quite confusing. The configurations actually mean something different fo= r local and remote environments:

=C2=A0 - For the local = environment, the config basically describes the entire Flink cluster setup = (for the local execution cluster in the background)
=C2=A0 - For = the remote environment, the config describes the parameters for the client = that connects to the cluster (akka paramters, optimizer parameters, ...), b= ut not parameters of the cluster itself (like taskmanager slots and memory)= .

Greetings,
Stephan


= On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <<= a href=3D"mailto:pompermaier@okkam.it" target=3D"_blank">pompermaier@okkam.= it> wrote:
However it co= uld be a good idea to overload also the=C2=A0getExecutionEnvironment() to b= e able to pass a custom configuration..what do you think?
Otherwise I h= ave to know a priori if I'm working in a local deployment or in a remot= e one, or check if=C2=A0getExecutionEnvironment() returned an instance of L= ocalEnvironment/RemoteEnvironment..


=

On Tue, Oct= 6, 2015 at 10:53 AM, Flavio Pompermaier <pompermaier@okkam.it><= /span> wrote:
Yes Stephan!
I usual= ly work with the master version, at least in development ;)
Thanks for = the quick support!

Best,
Flavio

On Tue, Oc= t 6, 2015 at 10:48 AM, Stephan Ewen <sewen@apache.org> wrote:=
Hi!

Are you on the= SNAPSHOT master version?

You can pass the configu= ration to the constructor of the execution environment, or create one via E= xecutionEnvironment.createLocalEnvironment(config) or via createRemoteEnvir= onment(host, port, configuration, jarFiles);

The c= hange of the signature was part of an API cleanup for the next release. Sor= ry for the inconvenience...

Stephan


On Tue, Oct 6, 2015 at 10:= 36 AM, Flavio Pompermaier <pompermaier@okkam.it> wrote:
Hi to all,

=

today my code doesn't compile anymore because=C2=A0ExecutionEnviro= nment doesn't have setConfiguration() anymore..how can I set the follow= ing parameters in my unit tests?

- ConfigConstants= .TASK_MANAGER_TMP_DIR_KEY
- ConfigConstants.BLOB_STORAGE_DIRE= CTORY_KEY
-=C2=A0ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS= _KEY

Best,
Flavio


<= /p>




<= /p>



--001a114fba5e634e99052213b3e7--