reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <>
Subject [jira] [Commented] (REEF-1224) IMRU Fault Tolerance - Separate Data downloading from Task injection
Date Tue, 15 Mar 2016 18:31:33 GMT


Markus Weimer commented on REEF-1224:

Not even that is needed. If {{IPartition.Get()}} hangs on to a reference of the data and returns
it on subsequent calls, life is good. A more advanced version would spawn a thread to do the
download / parse work in the constructor of {{IPartition}} and then return. {{IPartition.Get()}}
would then join that thread before returning. That way, we can report success in setting up
the {{ActiveContext}} to the Driver and overlap the actual data loading with the latency for
the Driver to get back to us with the Task to run.

> IMRU Fault Tolerance - Separate Data downloading from Task injection
> --------------------------------------------------------------------
>                 Key: REEF-1224
>                 URL:
>             Project: REEF
>          Issue Type: Improvement
>          Components: IMRU, REEF.NET
>            Reporter: Julia
>            Assignee: Dhruv Mahajan
> Currently in IMRU, data downloading happens during the Task injection. It couples the
data and Task object. In Fault tolerant case, we would like to only resubmit a task but use
the data that have been downloaded, That requires us to decouple those two portions. For example,
data downloading portion can be attached to Context, and we can then resubmit a task on the
same context. 

This message was sent by Atlassian JIRA

View raw message