Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Tue, 22 Nov 2016 19:54:22 +0100
From: fschueler@posteo.de
To: dev@systemml.incubator.apache.org
Subject: Re: Parfor semantics
In-Reply-To: <577b77b8-8a66-620b-ceca-ff24a04041ef@gmail.com>
References: <4f097e10a8e5734af4494daaa138b17d@posteo.de>
 <577b77b8-8a66-620b-ceca-ff24a04041ef@gmail.com>
Message-ID: <460842a0fdc82e9bb327b7fb0689d8fb@posteo.de>
User-Agent: Posteo Webmail
archived-at: Tue, 22 Nov 2016 18:54:37 -0000

The constrained optimizer doesn't seem to know about a REMOTE_SPARK 
execution mode and either sets CP or REMOTE_MR. I can open a jira for 
that and provide a fix.

Felix

Am 22.11.2016 02:07 schrieb Matthias Boehm:
> yes, this came up several times - initially we only supported opt=NONE
> where users had to specify all other parameters. Meanwhile, there is a
> so-called "constrained optimizer" that does the same as the rule-based
> optimizer but respects any given parameters. Please try something like
> this:
> 
> parfor (i in 1:10, opt=CONSTRAINED, par=10, mode=REMOTE_SPARK) {
>      // some code here
> }
> 
> 
> Regards,
> Matthias
> 
> On 11/22/2016 12:33 AM, fschueler@posteo.de wrote:
>> While debugging some ParFor code it became clear that the parameters 
>> for
>> parfor can be easily overwritten by the optimizer.
>> One example is when I write:
>> 
>> ```
>> parfor (i in 1:10, par=10, mode=REMOTE_SPARK) {
>>     // some code here
>> }
>> ```
>> 
>> Depending on the data size and cluster resources, the optimizer
>> (OptimizerRuleBased.java, line 844) will recognize that the work can 
>> be
>> done locally and overwrite it to local execution. This might be valid
>> and definitely works (in my case) but kind of contradicts what I want
>> SystemML to do.
>> I wonder if we should disable this optimization in case a concrete
>> execution mode is given and go with the mode that is provided.
>> 
>> Felix
>> 
>>