reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruv Mahajan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (REEF-1224) IMRU Fault Tolerance - Separate Data downloading from Task injection
Date Fri, 18 Mar 2016 18:52:33 GMT

    [ https://issues.apache.org/jira/browse/REEF-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201959#comment-15201959
] 

Dhruv Mahajan edited comment on REEF-1224 at 3/18/16 6:52 PM:
--------------------------------------------------------------

So I started implementing and realised that since {{OnContextStart}} needs to instantiate
{{IInputPartition<T>}}, the dataset handler type T now needs to be known beforehand
to driver. Hence in addition to {{TMapInput}}, {{TMapOutput}} and {{TResult}}, we need to
expose another Generic Type i.e. {{TDataHandlerType}}.

I could not think of any workaround. Any suggestions?




was (Author: dkm2110):
So I started implementing and realised that since OnContextStart needs to instantiate IInputPartition<T>,
the dataset handler type T now needs to be known beforehand to driver. Hence in addition to
TMapInput, TMapOutput,TResult we need to expose another Generic Type i.e. TDataHandlerType.



> IMRU Fault Tolerance - Separate Data downloading from Task injection
> --------------------------------------------------------------------
>
>                 Key: REEF-1224
>                 URL: https://issues.apache.org/jira/browse/REEF-1224
>             Project: REEF
>          Issue Type: Improvement
>          Components: IMRU, REEF.NET
>            Reporter: Julia
>            Assignee: Dhruv Mahajan
>
> Currently in IMRU, data downloading happens during the Task injection. It couples the
data and Task object. In Fault tolerant case, we would like to only resubmit a task but use
the data that have been downloaded, That requires us to decouple those two portions. For example,
data downloading portion can be attached to Context, and we can then resubmit a task on the
same context. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message