systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From fschue...@posteo.de
Subject Re: Parfor semantics
Date Tue, 22 Nov 2016 18:54:22 GMT
The constrained optimizer doesn't seem to know about a REMOTE_SPARK 
execution mode and either sets CP or REMOTE_MR. I can open a jira for 
that and provide a fix.

Felix

Am 22.11.2016 02:07 schrieb Matthias Boehm:
> yes, this came up several times - initially we only supported opt=NONE
> where users had to specify all other parameters. Meanwhile, there is a
> so-called "constrained optimizer" that does the same as the rule-based
> optimizer but respects any given parameters. Please try something like
> this:
> 
> parfor (i in 1:10, opt=CONSTRAINED, par=10, mode=REMOTE_SPARK) {
>      // some code here
> }
> 
> 
> Regards,
> Matthias
> 
> On 11/22/2016 12:33 AM, fschueler@posteo.de wrote:
>> While debugging some ParFor code it became clear that the parameters 
>> for
>> parfor can be easily overwritten by the optimizer.
>> One example is when I write:
>> 
>> ```
>> parfor (i in 1:10, par=10, mode=REMOTE_SPARK) {
>>     // some code here
>> }
>> ```
>> 
>> Depending on the data size and cluster resources, the optimizer
>> (OptimizerRuleBased.java, line 844) will recognize that the work can 
>> be
>> done locally and overwrite it to local execution. This might be valid
>> and definitely works (in my case) but kind of contradicts what I want
>> SystemML to do.
>> I wonder if we should disable this optimization in case a concrete
>> execution mode is given and go with the mode that is provided.
>> 
>> Felix
>> 
>> 


Mime
View raw message