reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruv Mahajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1224) IMRU Fault Tolerance - Separate Data downloading from Task injection
Date Tue, 22 Mar 2016 18:40:25 GMT

    [ https://issues.apache.org/jira/browse/REEF-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206997#comment-15206997
] 

Dhruv Mahajan commented on REEF-1224:
-------------------------------------

Puttign the email thread related to the JIRA here. Please read bottom to top.

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 9:26 AM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Hi,

I filed https://issues.apache.org/jira/browse/REEF-1268 for this.

Markus

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 09:22
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Right, that should be added. The Java equivalent has it. I’ll file a JIRA.

Markus

From: Dhruv Mahajan 
Sent: Tuesday, March 22, 2016 09:17
To: Markus Weimer <mweimer@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

As far as I know, ServiceConfiguration has no in-built method to bind IObserver<OnContextStart>.


Dhruv

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 9:14 AM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Let’s be more precise: Use the class `ServiceConfiguration` to generate the service configuration,
not `ContextConfiguration`.

Markus 

From: Dhruv Mahajan 
Sent: Tuesday, March 22, 2016 09:11
To: Markus Weimer <mweimer@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Yes I will use both….What I am saying is that I cannot use the inbuilt ContextConfiguration
module for specifying service configuration  since providing Id is mandatory there and we
shud not do it in service conf.

Dhruv

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 9:04 AM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

You can use both a Context and a Service Configuration, right? The ID goes to the context,
the data to the service configuration.

Markus

From: Dhruv Mahajan 
Sent: Tuesday, March 22, 2016 09:03
To: Markus Weimer <mweimer@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Ok, just to complete the thread and then I will put the discussion on JIRA, I cannot use ContextConfiguration
module here since ID is must otherwise.

Dhruv

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 9:01 AM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Yes, I understand that. However, other people will make the same mistakes or have the same
questions. Keeping these threads private robs us of a documentation opportunity.

Markus

From: Dhruv Mahajan 
Sent: Tuesday, March 22, 2016 08:58
To: Markus Weimer <mweimer@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Sure boss…..I thought it was simple error from my side…that’s why  ….but will put
it there 

Dhruv

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 8:56 AM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Obligatory comment from the PMC chair: We again have half the conversation in public and half
here. 

Why can’t you call SubmitContextAndService() with a context config with the ID and the service
config with the handler? That’s how it is intended to be used.

Markus

From: Dhruv Mahajan 
Sent: Tuesday, March 22, 2016 08:52
To: Markus Weimer <mweimer@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

The problem is context configuration cannot be empty it needs atleast one context with some
Id.
Now if I put contextStart in Services, I  have to put Id there which then leads to Tang error
of same key exists,

Dhruv

From: Markus Weimer 
Sent: Tuesday, March 22, 2016 8:49 AM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

Hi,

You could also just make a ContextStart handler that depends on this type. That way, you don’t
have to interact with the `ServiceConfiguration.Services` configuration option, which we are
likely deprecating anyhow as per the long threads on services on the dev list.

Markus


From: Dhruv Mahajan 
Sent: Tuesday, March 22, 2016 08:41
To: Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>; Markus Weimer <mweimer@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

IS there something witn with this? TDataHandler is Generic Type

ServiceConfiguration.ConfigurationModule.Set(ServiceConfiguration.Services,
                        GenericType<IInputPartition<TDataHandler>>.Class).Build()

From: Julia Wang (QIUHE) 
Sent: Monday, March 21, 2016 11:53 PM
To: Dhruv Mahajan <dhrumaha@microsoft.com>; Markus Weimer <mweimer@microsoft.com>
Subject: RE: Issues with ServiceConfiguration

There shouldn’t have any cross language deserialization issue here. We have by passed that
portion in Java. 

Looks like some issue in your service configuration. 

From: Dhruv Mahajan 
Sent: Monday, March 21, 2016 11:23 PM
To: Markus Weimer <mweimer@microsoft.com>; Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com>
Subject: Issues with ServiceConfiguration

Hi,

I binded IInputPartition<TDataHandler> as Service in ServiceConfiguration class. However,
on evaluator side I got the following error when it was starting. I am highlighting the relevant
portion. Is it because of the same issue we faced earlier when we were passing generics to
the interface and on the other side deserialization of configuration was becoming an issue?

Org.Apache.REEF.Common.Protobuf.ReefProtocol.REEFMessageCodec(), [Org.Apache.REEF.Tang.Implementations.ClassHierarchy.ClassNodeImpl
'Org.Apache.REEF.Tang.Interface.IInjector, Org.Apache.REEF.Tang, Version=0.15.0.0, Culture=neutral,
PublicKeyToken=c27bf5b2e9a7ddb9']:  = Org.Apache.REEF.Tang.Implementations.InjectionPlan.InjectorImpl)
---> System.Reflection.TargetInvocationException: Exception has been thrown by the target
of an invocation. ---> Org.Apache.REEF.Tang.Exceptions.BindException: Failed to process
configuration tuple: [Org.Apache.REEF.Common.Services.ServicesSet, Org.Apache.REEF.Common,
Version=0.15.0.0, Culture=neutral, PublicKeyToken=c27bf5b2e9a7ddb9=Org.Apache.REEF.IO.PartitionedData.IInputPartition`1[[System.IO.Stream,
mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Org.Apache.REEF.IO,
Version=0.15.0.0, Culture=neutral, PublicKeyToken=c27bf5b2e9a7ddb9] ---> Org.Apache.REEF.Tang.Exceptions.ParseException:
Name<System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089>
Org.Apache.REEF.Common.Services.ServicesSet, Org.Apache.REEF.Common, Version=0.15.0.0, Culture=neutral,
PublicKeyToken=c27bf5b2e9a7ddb9 cannot take non-subclass Org.Apache.REEF.IO.PartitionedData.IInputPartition`1[[System.IO.Stream,
mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Org.Apache.REEF.IO,
Version=0.15.0.0, Culture=neutral, PublicKeyToken=c27bf5b2e9a7ddb9
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Org.Apache.REEF.Tang.Implementations.ClassHierarchy.ClassHierarchyImpl.Parse(INamedParameterNode
np, String value)
   at Org.Apache.REEF.Tang.Implementations.Configuration.ConfigurationBuilderImpl.BindParameter(INamedParameterNode
name, String value)
   at Org.Apache.REEF.Tang.Formats.ConfigurationFile.ProcessConfigData(IConfigurationBuilder
conf, IList`1 settings, String language)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Org.Apache.REEF.Tang.Formats.ConfigurationFile.ProcessConfigData(IConfigurationBuilder
conf, IList`1 settings, String language)
   at Org.Apache.REEF.Tang.Formats.AvroConfigurationSerializer.AddFromAvro(IConfigurationBuilder
cb, AvroConfiguration avroConfiguration)
   at Org.Apache.REEF.Common.Runtime.Evaluator.EvaluatorSettings.CreateRootServiceConfiguration()
   at Org.Apache.REEF.Common.Runtime.Evaluator.EvaluatorSettings..ctor(String applicationId,
String evaluatorId, Int32 heartbeatPeriodInMs, Int32 maxHeartbeatRetries, String rootContextConfigString,
AvroConfigurationSerializer serializer, RuntimeClock clock, IRemoteManagerFactory remoteManagerFactory,
REEFMessageCodec reefMessageCodec, IInjector injector, INameClient nameClient)
   at Org.Apache.REEF.Common.Runtime.Evaluator.EvaluatorSettings..ctor(String applicationId,
String evaluatorId, Int32 heartbeatPeriodInMs, Int32 maxHeartbeatRetries, String rootContextConfigString,
AvroConfigurationSerializer serializer, RuntimeClock clock, IRemoteManagerFactory remoteManagerFactory,
REEFMessageCodec reefMessageCodec, IInjector injector)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature
sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder,
Object[] parameters, CultureInfo culture)
   at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
   at Org.Apache.REEF.Tang.Implementations.InjectionPlan.InjectorImpl.InjectFromPlan(InjectionPlan
plan)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Org.Apache.REEF.Tang.Implementations.InjectionPlan.InjectorImpl.InjectFromPlan(InjectionPlan
plan)
   at Org.Apache.REEF.Tang.Implementations.InjectionPlan.InjectorImpl.InjectFromPlan(InjectionPlan
plan)
   at Org.Apache.REEF.Tang.Implementations.InjectionPlan.InjectorImpl.InjectFromPlan(InjectionPlan
plan)


> IMRU Fault Tolerance - Separate Data downloading from Task injection
> --------------------------------------------------------------------
>
>                 Key: REEF-1224
>                 URL: https://issues.apache.org/jira/browse/REEF-1224
>             Project: REEF
>          Issue Type: Improvement
>          Components: IMRU, REEF.NET
>            Reporter: Julia
>            Assignee: Dhruv Mahajan
>
> Currently in IMRU, data downloading happens during the Task injection. It couples the
data and Task object. In Fault tolerant case, we would like to only resubmit a task but use
the data that have been downloaded, That requires us to decouple those two portions. For example,
data downloading portion can be attached to Context, and we can then resubmit a task on the
same context. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message