systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SYSTEMML-1392) Redundant parfor spark dpe result var export
Date Tue, 02 May 2017 19:00:05 GMT

     [ https://issues.apache.org/jira/browse/SYSTEMML-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Glenn Weidner updated SYSTEMML-1392:
------------------------------------
    Fix Version/s:     (was: SystemML 1.0)
                   SystemML 0.14

> Redundant parfor spark dpe result var export
> --------------------------------------------
>
>                 Key: SYSTEMML-1392
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1392
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 0.14
>
>
> The parfor spark datapartition-execute job current writes result variables per parfor
input partition. However, since a reduce task likely has multiple parfor partitions and outputs
are guaranteed to have no conflicts, this leads to unnecessary write overhead. 
> To fix this issues, we should only write result variables once per physical partition.
Similarly, since accumulators are only reported for finished tasks, we should also maintain
these task/iteration accumulators just once per task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message