reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1842) Making IMRU task and input data association deterministic
Date Wed, 02 Aug 2017 18:27:00 GMT

    [ https://issues.apache.org/jira/browse/REEF-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111476#comment-16111476
] 

Julia commented on REEF-1842:
-----------------------------

Not user supplied mapping. The input data file descriptors can be still randomly given. Internally
we will make them have a fix mapping with contexid, so that the data files assigned to tasks
on topology are always deterministic. So yes, it is still random but deterministic. 

> Making IMRU task and input data association deterministic
> ---------------------------------------------------------
>
>                 Key: REEF-1842
>                 URL: https://issues.apache.org/jira/browse/REEF-1842
>             Project: REEF
>          Issue Type: Task
>            Reporter: Julia
>            Assignee: Julia
>            Priority: Critical
>
> Currently, the tasks added to the typology on communication group is based on the sequence
of contexts received. While the input data assigned to context can be random especially when
some evaluators fail and new evaluators/contexts are requested. This makes the training result
undeterministic given same input data set. 
> To make the result deterministic for given set of input data, we would like to make tasks
in the typology keep the same order with the same input data file associated. To achieve it,
the mapping between the context id and partitionDescriptor id should be fixed once assigned.
And when adding tasks to the communication group, the order should be based on the sorted
id to ensure it is deterministic. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message