systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LI Guobao (JIRA)" <>
Subject [jira] [Created] (SYSTEMML-2465) Keep data consistency for a pre-trained model
Date Wed, 25 Jul 2018 20:57:00 GMT
LI Guobao created SYSTEMML-2465:

             Summary: Keep data consistency for a pre-trained model
                 Key: SYSTEMML-2465
             Project: SystemML
          Issue Type: Sub-task
            Reporter: LI Guobao
            Assignee: LI Guobao

In distributed spark backend, pass a given pre-trained model to the paramserv function may
cause the data inconsistency. Because the pre-trained model would be cached in driver's memory.
In this case, when kicking off the paramserv func, the workers firstly will try to read the
data from HDFS where the dirty data in pre-trained model has not been persisted. This leads
to a inconsistency. So the idea is to export the dirty data to HDFS before kicking off the
remote workers.

This message was sent by Atlassian JIRA

View raw message