systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (SYSTEMML-2163) Performance large partitioned broadcasts
Date Thu, 01 Mar 2018 05:31:00 GMT

     [ https://issues.apache.org/jira/browse/SYSTEMML-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matthias Boehm closed SYSTEMML-2163.
------------------------------------
       Resolution: Fixed
         Assignee: Matthias Boehm
    Fix Version/s: SystemML 1.1

> Performance large partitioned broadcasts
> ----------------------------------------
>
>                 Key: SYSTEMML-2163
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2163
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>            Priority: Major
>             Fix For: SystemML 1.1
>
>
> Due to Spark's limitations with broadcasts larger than 2G, in SystemML we use partitioned
broadcasts that split a large side input into potentially many broadcast variables. For historic
reasons the meta data is still maintained in the individual partitioned blocks. However, for
many operations this meta data is accessed on the first partitioned block which leads to potentially
unnecessary broadcast fetches.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message