systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner (JIRA)" <>
Subject [jira] [Updated] (SYSTEMML-1392) Redundant parfor spark dpe result var export
Date Tue, 02 May 2017 19:00:05 GMT


Glenn Weidner updated SYSTEMML-1392:
    Fix Version/s:     (was: SystemML 1.0)
                   SystemML 0.14

> Redundant parfor spark dpe result var export
> --------------------------------------------
>                 Key: SYSTEMML-1392
>                 URL:
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 0.14
> The parfor spark datapartition-execute job current writes result variables per parfor
input partition. However, since a reduce task likely has multiple parfor partitions and outputs
are guaranteed to have no conflicts, this leads to unnecessary write overhead. 
> To fix this issues, we should only write result variables once per physical partition.
Similarly, since accumulators are only reported for finished tasks, we should also maintain
these task/iteration accumulators just once per task.

This message was sent by Atlassian JIRA

View raw message