reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Chung (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (REEF-1412) Use a two-file approach for DFSEvaluatorLogOverwriteWriter
Date Thu, 02 Jun 2016 18:49:59 GMT

     [ https://issues.apache.org/jira/browse/REEF-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Chung updated REEF-1412:
-------------------------------
    Description: 
A problem that needs to be addressed is that it currently deletes the old EvaluatorChangesLog
prior to writing the new one. Instead, we should use a two-file approach, where an alternative
file, EvaluatorChangesLog.alt, is used in tandem with EvaluatorChangesLog.
On read, the newest between EvaluatorChangesLog and EvaluatorChangesLog.alt should always
be read. 
On write, we always overwrite the older file between EvaluatorChangesLog and EvaluatorChangesLog.alt.




  was:
{{DFSEvaluatorLogOverwriteWriter}} currently reads from original EvalutorChangesLog before
overwriting it. This can become a problem when the job has been running for a long time. We
can make this more efficient by simply recording the Evaluators it expects in a log.

Another problem that needs to be addressed is that it currently deletes the old EvaluatorChangesLog
prior to writing the new one. Instead, we should use a two-file approach, where an alternative
file, EvaluatorChangesLog.alt, is used in tandem with EvaluatorChangesLog.
On read, the newest between EvaluatorChangesLog and EvaluatorChangesLog.alt should always
be read. 
On write, we always overwrite the older file between EvaluatorChangesLog and EvaluatorChangesLog.alt.





> Use a two-file approach for DFSEvaluatorLogOverwriteWriter
> ----------------------------------------------------------
>
>                 Key: REEF-1412
>                 URL: https://issues.apache.org/jira/browse/REEF-1412
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF Driver, REEF.NET Driver
>            Reporter: Andrew Chung
>            Assignee: Andrew Chung
>
> A problem that needs to be addressed is that it currently deletes the old EvaluatorChangesLog
prior to writing the new one. Instead, we should use a two-file approach, where an alternative
file, EvaluatorChangesLog.alt, is used in tandem with EvaluatorChangesLog.
> On read, the newest between EvaluatorChangesLog and EvaluatorChangesLog.alt should always
be read. 
> On write, we always overwrite the older file between EvaluatorChangesLog and EvaluatorChangesLog.alt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message