ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Sapego <isap...@gridgain.com>
Subject Re: Remote job execution in C++
Date Mon, 26 Oct 2015 16:06:21 GMT
I don't see any problems in dynamic jobs loading with current approach.

As far as I can see the only thing we are going to need to be able to
load new library without restarting the cluster is some way to tell nodes
which library they should load.

Also we will need to be able to revert the operation if one or more nodes
were not able to load a library.

At least this is how I see it from the point of view of C++ components.


Best Regards,
Igor

On Mon, Oct 26, 2015 at 6:23 PM, Yakov Zhdanov <yzhdanov@apache.org> wrote:

> Agree, I am talking about some future releases.
>
> --Yakov
>
> 2015-10-26 14:06 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
>
> > Job load without cluster restart is advanced feature. I think we should
> > make basic API work first.
> >
> > On Mon, Oct 26, 2015 at 12:16 PM, Yakov Zhdanov <yzhdanov@apache.org>
> > wrote:
> >
> > > I think suggested approach should work.
> > >
> > > Any idea on how to add dynamic jobs loading (I would like it is added
> in
> > > one of nearest releases)? I.e. I copy new library to each server then
> > load
> > > it with Ignite API and have new jobs available without restarting
> > cluster.
> > >
> > > --Yakov
> > >
> > > 2015-10-20 18:46 GMT+03:00 Igor Sapego <isapego@gridgain.com>:
> > >
> > > > Hello Guys,
> > > >
> > > > Recently I've been working on the IGNITE-1680: CPP: Implement basic
> API
> > > for
> > > > remote job execution
> https://issues.apache.org/jira/browse/IGNITE-1680
> > .
> > > >
> > > > I would like to discuss the approach I came up with.
> > > >
> > > > In my current approach I use existing portable mechanism to serialize
> > the
> > > > job object so we can send it to the remote node. However in C++
> unlike
> > in
> > > > Java or C# we do not have any out-of-box way to instantiate objects
> of
> > > type
> > > > which is unknown on the compile time. Still we have some runtime type
> > > > information provided by the portable layer (e.g. we have type id). So
> > we
> > > > need to implement solution to deserialize and invoke a job object and
> > > > serialize result of this invocation. To put it simple we are going to
> > > need
> > > > something like this:
> > > >
> > > > bool InvokeRemoteJob(int32_t jobTypeId,
> > > >                      PortableReaderImpl& jobReader,
> > > >                      PortableWriterImpl& resultWriter);
> > > >
> > > > As we can not operate with type during runtime in C++ it is proposed
> to
> > > > make user to provide us with such kind of function. We can define
> three
> > > > simple macros so user can to provide us with the list of the jobs
> like
> > > > this:
> > > >
> > > > IGNITE_REMOTE_JOB_LIST_BEGIN
> > > > IGNITE_REMOTE_JOB_DECLARE(JobType1, ReturnType1)
> > > > IGNITE_REMOTE_JOB_DECLARE(JobType2, ReturnType2)
> > > > ...
> > > > IGNITE_REMOTE_JOB_DECLARE(JobTypeN, ReturnTypeN)
> > > > IGNITE_REMOTE_JOB_LIST_END
> > > >
> > > > This will expand to something like that:
> > > >
> > > > extern "C" IGNITE_IMPORT_EXPORT
> > > > bool InvokeRemoteJob(int32_t jobTypeId,
> > > > PortableReaderImpl& jobReader,
> > > > PortableWriterImpl& resultWriter)
> > > > {
> > > > if (jobTypeId == PortableType<JobType1>::GetTypeId())
> > > > {
> > > > JobType1 job(jobReader.ReadTopObject<JobType1>());
> > > > ReturnType1 call_res = job.call();
> > > > resultWriter.WriteTopObject(call_res);
> > > > return true;
> > > > }
> > > > ...
> > > > return false;
> > > > }
> > > >
> > > > Every module (dynamic library or executable) that has jobs that could
> > be
> > > > invoked remotely is going to need to have list of the remote jobs
> > > defined.
> > > > Then while we are loading modules we can lookup InvokeRemoteJob
> > function
> > > > for every module and store them so we can invoke jobs later.
> > > >
> > > > I have implemented proof of concept for this approach and proved it
> > > > working. Now I would like to hear what do you people think about it
> so
> > > > please provide your comments if any.
> > > >
> > > > Thanks
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message