systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <>
Subject [jira] [Updated] (SYSTEMML-1309) Parfor spark buffer pool handling
Date Fri, 21 Apr 2017 05:03:04 GMT


Matthias Boehm updated SYSTEMML-1309:
    Fix Version/s:     (was: SystemML 1.0)
                   SystemML 0.14

> Parfor spark buffer pool handling
> ---------------------------------
>                 Key: SYSTEMML-1309
>                 URL:
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: APIs, Runtime
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 0.14
> In contrast to parfor mr jobs, where every task has its own, process-local buffer pool,
on spark with multi-threaded executors, multiple tasks share a common buffer pool. This is
advantageous because common inputs are just read once. However, it also requires a synchronized
buffer pool initialization and cleanup per executor. Especially the cleanup (e.g., of created
cache directories) is tricky because spark does not provide an executor close call. Hence,
our approach is to use a robust version of deleteOnExit that is independent of the exit code
and also removes remaining files that are unknown during delete registration.  

This message was sent by Atlassian JIRA

View raw message