reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1224) IMRU Fault Tolerance - Separate Data downloading from Task injection
Date Tue, 15 Mar 2016 17:01:33 GMT

    [ https://issues.apache.org/jira/browse/REEF-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195665#comment-15195665
] 

Markus Weimer commented on REEF-1224:
-------------------------------------

I don't think we have good code examples for this, given that it only started working very
recently.

A simple way to implement this is to move the {{Configuration}} returned from {{IPartitionDescriptor.GetPartitionConfiguration()}}
into the *Service* configuration. According to line 157 of {{ServiceAndContextConfigurationProvider.cs}},
this is already the case:

{code}
var serviceConf = Configurations.Merge(_groupCommDriver.GetServiceConfiguration(), codecConfig,
partitionDescriptor.GetPartitionConfiguration());
{code}

Now, the only thing we need to make sure is that the {{IPartition}} instance is generated
on the service injector instead of the Task injector. One way to do that is to add a new {{private
readonly IConfiguration _partitionConfiguration}} to {{ServiceAndContextConfigurationProvider}}
and then merge that in {{ServiceAndContextConfigurationProvider.GetMapTaskContextAndServiceConfiguration}}.
{{_partitionConfiguration}} could be defined as:

{code}
_partitionConfiguration = ContextConfiguration.ConfigurationModule.Set(ContextConfiguration.OnContextStart,
...)
{code}

The {{...}} would have to be replaced with a simple context start handler that depends on
{{IPartition<T>}}.

This should do the basic trick. Beyond that, we probably need the ability to reassign a partition
inside of {{ServiceAndContextConfigurationProvider}}, which currently works off of {{evaluatorId}}s.
If an Evaluator fails, this is probably not the right way to do it. Maybe we need to instead
work off of partition IDs and maintain a mapping of which Evaluator is assigned which partition.

> IMRU Fault Tolerance - Separate Data downloading from Task injection
> --------------------------------------------------------------------
>
>                 Key: REEF-1224
>                 URL: https://issues.apache.org/jira/browse/REEF-1224
>             Project: REEF
>          Issue Type: Improvement
>          Components: IMRU, REEF.NET
>            Reporter: Julia
>            Assignee: Dhruv Mahajan
>
> Currently in IMRU, data downloading happens during the Task injection. It couples the
data and Task object. In Fault tolerant case, we would like to only resubmit a task but use
the data that have been downloaded, That requires us to decouple those two portions. For example,
data downloading portion can be attached to Context, and we can then resubmit a task on the
same context. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message