systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Commented] (SYSTEMML-1313) Parfor broadcast exploitation
Date Fri, 13 Apr 2018 19:56:00 GMT


Matthias Boehm commented on SYSTEMML-1313:

thanks for your interest [~Guobao] - I updated the description accordingly. I would recommend
to introduce the runtime support first (temporarily enabled for testing, but disabled by default).

> Parfor broadcast exploitation
> -----------------------------
>                 Key: SYSTEMML-1313
>                 URL:
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: APIs, Runtime
>            Reporter: Matthias Boehm
>            Priority: Major
> The parfor optimizer may decide to execute the entire loop as a remote Spark job to utilize
cluster parallelism. In this case all inputs to the parfor body (i.e., variable that are created
or read outside of the parfor body but used or overwritten inside) are read from HDFS. In
the past there was an issue of redundant reads, which has been addressed with SYSTEMML-1879.
However, the direct use of Spark broadcast variables would likely improve performance, especially
in clusters with many nodes.
> This task aims to leverage Spark broadcast variables for all parfor inputs. In detail
this entails two major aspects. First, we need runtime support to optionally broadcast the
inputs via broadcast variables in {{RemoteParForSpark}} and obtain them from these broadcast
variables in {{RemoteParForSparkWorker}} without causing unnecessary eviction. In contrast,
to the existing broadcast primitives, we don't need to blockify the matrix because the matrix
is accessed in full by in-memory operations. Second, this requires an extension of the parfor
optimizer to reason about scenarios where it is safe to use broadcast because these broadcasts
cause additional memory requirements since they act as pinned in memory matrices. This second
task has likely overlap with SYSTEMML-1349 which requires a similar reasoning to handle shared

This message was sent by Atlassian JIRA

View raw message