ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Алексей Платонов <aplaton...@gmail.com>
Subject Re: [ML] Deployment of user-defined preprocessors
Date Fri, 31 May 2019 12:33:20 GMT
No, I won't change them.
In context of this architecture we should pass learning environment to
ComputeUtils and this concerns just DatasetBuilders and ComputeUtils APIs.
APIs of Preprocessors and Trainers won't be changed. We just set
DeploymentContext in fit method but it doens't require to change APIs of
these classes.

Sincerely
Alexey Platonov

пт, 31 мая 2019 г. в 14:51, Alexey Zinoviev <zaleslaw.sin@gmail.com>:

> It sounds great, could you please explain what are you going to change in
> the main Vectorizer/Preprocessor API to support it?
>
> пт, 31 мая 2019 г. в 14:20, Алексей Платонов <aplatonovv@gmail.com>:
>
> > Hi, Igniters!
> > Currently we don't have an ability to deploy automatically user-defined
> > preprocessors and vectorizers. Client's code should be deployed manually
> to
> > Ignite server nodes.
> >
> > I have an idea how to fix it. If we pass user's classloader and one of
> > user-defined classes from fit-level to
> > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > GridPeerDeployAware interface to send informtation about this classloader
> > to server nodes.
> >
> > To support this ability we can define interfaces like these:
> >
> > public interface DeployableObject {
> >     public List<Object> getDependencies();
> > }
> >
> > and
> >
> > public interface DeployingContext {
> >     public Class<?> userClass();
> >     public ClassLoader clientClassLoader();
> > }
> >
> > DeployableObject will be mark for our ignite-ml final classes like
> trainers
> > or concrete preprocessors and it can be able to return all dependencies
> > that should be deployed to server nodes if it's needed. If these
> > dependencies are DeployableObjects too then depenndencies will be
> unfolded
> > recursively. Classes that isn't defined as DeployableObject will be
> > recognized as user-defined (NOTE: all leaf classes in our hierarchy will
> be
> > DeployableObject).
> >
> > This list of DeployableObjects will be user for define user class loader
> > and one of these objects will be used for passing to GridPeerDeployAware.
> >
> > So, this logic allows to pass user-defined Preprocessors and Vectorizers
> to
> > training algorithms and pipelines.
> >
> > What do you think?
> >
> > Sincerely
> > Alexey Platonov
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message