systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Closed] (SYSTEMML-2021) Perftest stratstats w/ codegen getting stuck in optimization
Date Sun, 19 Nov 2017 21:57:00 GMT


Matthias Boehm closed SYSTEMML-2021.
       Resolution: Fixed
         Assignee: Matthias Boehm
    Fix Version/s: SystemML 1.0

> Perftest stratstats w/ codegen getting stuck in optimization
> ------------------------------------------------------------
>                 Key: SYSTEMML-2021
>                 URL:
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
> On stratstats 10K, the codegen optimizer gets stuck during plan enumeration of a DAG
partition with 274,877,906,944 plans. During initial compilation, the same partition was processed
in 2.8s as most candidate plans have been successfully pruned by cost or structure.
> After a detailed analysis, we can characterize the problem as follows: there is a fusion
partition with 38 interesting points, resulting in the huge search space of 274 billion plans.
Furthermore, all these materialization points are extremely small while at the same time there
are large inputs and compute-intensive operations as well as many common subexpression, which
renders the pruning by costs starting from the fuse-all heuristic ineffective. 
> We can address this as follows: First, we should run both heuristics (fuse-all and fuse-no-redundancy)
which are the first and last plan upfront to quickly obtain a good lower bound for the costs.
Second, we should only consider enumerating large partitions (say with more than 20 interesting
points) if the opening heuristics show costs that is further than an epsilon (say 1%) away
from the minimal static costs. Together these rules will make the optimizer significantly
more robust without missing any meaningful plan choices.

This message was sent by Atlassian JIRA

View raw message