reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Chung (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (REEF-1412) Use a two-file approach for DFSEvaluatorLogOverwriteWriter
Date Thu, 02 Jun 2016 18:49:59 GMT

     [ https://issues.apache.org/jira/browse/REEF-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Chung updated REEF-1412:
-------------------------------
    Summary: Use a two-file approach for DFSEvaluatorLogOverwriteWriter  (was: Improve efficiency
of DFSEvaluatorLogOverwriteWriter)

> Use a two-file approach for DFSEvaluatorLogOverwriteWriter
> ----------------------------------------------------------
>
>                 Key: REEF-1412
>                 URL: https://issues.apache.org/jira/browse/REEF-1412
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF Driver, REEF.NET Driver
>            Reporter: Andrew Chung
>            Assignee: Andrew Chung
>
> {{DFSEvaluatorLogOverwriteWriter}} currently reads from original EvalutorChangesLog before
overwriting it. This can become a problem when the job has been running for a long time. We
can make this more efficient by simply recording the Evaluators it expects in a log.
> Another problem that needs to be addressed is that it currently deletes the old EvaluatorChangesLog
prior to writing the new one. Instead, we should use a two-file approach, where an alternative
file, EvaluatorChangesLog.alt, is used in tandem with EvaluatorChangesLog.
> On read, the newest between EvaluatorChangesLog and EvaluatorChangesLog.alt should always
be read. 
> On write, we always overwrite the older file between EvaluatorChangesLog and EvaluatorChangesLog.alt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message