airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amila Jayasekara <thejaka.am...@gmail.com>
Subject Re: Orchestration Component implementation review
Date Mon, 20 Jan 2014 04:48:57 GMT
On Sun, Jan 19, 2014 at 5:33 PM, Lahiru Gunathilake <glahiru@gmail.com>wrote:

> Hi Chathuri,
>
>
> On Fri, Jan 17, 2014 at 11:40 AM, Chathuri Wimalasena <
> kamalasini@gmail.com> wrote:
>
>> Orchestrator table has only the current state (updated state). Previous
>> statuses should be saved in the GFac_Job_Status table.
>>
> Since the order of the steps are defined, do we need to store the previous
> states ?
>

I am also curious to know why we need to store previous states. We have a
define state diagram for a job. Also we have log files if we want to debug
a specific issue related to job statuses. So I am not sure why we need to
store previous job states. Also who and when we will access previous job
states ?

Thanks
Amila


>
> Regards
> Lahiru
>
>>
>> Regards,
>> Chathuri
>>
>>
>> On Fri, Jan 17, 2014 at 11:25 AM, Sachith Withana <swsachith@gmail.com>wrote:
>>
>>> Thanks Saminda for this informative review.
>>>
>>> In the case of the multi-threaded vs Single Threaded, where should we
>>> have the synchronization enforced?
>>> To my knowledge, the NewJobWorkers( Getting new Jobs and submitting
>>> them) and the HangedJobWorkers are accessing the Orchestrator table to
>>> select the new and hanged jobs.
>>> Right now, the NewJobworkers are getting all the accepted jobs at once.
>>> it's not focussed on one experiment.
>>>
>>> We need to reflect the changes in the Gfac job Statuses in the
>>> Orchestrator table as well. So every time the status of a job change
>>> through the Gfac, it will be accessing the Orchestrator table as well. (
>>> I've sent an email previously describing the scenario)
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jan 17, 2014 at 10:32 AM, Saminda Wijeratne <samindaw@gmail.com>wrote:
>>>
>>>> Following are few thoughts I had during my review of the component,
>>>>
>>>> *Multi-threaded vs single threaded*
>>>> If we are going to have multi-threaded job submission the
>>>> implementation should work on handling race conditions. Essentially
>>>> JobSubmitter should be able to "lock" an experiment request before
>>>> continuing processing that request so that other JobSubmitters accessing
>>>> the experiment requests a the same time would skip it.
>>>>
>>>> *Orchestrator service*
>>>> We might want to think of the possibility in future where we will be
>>>> having multiple deployments of an Airavata service. This could particularly
>>>> be true for SciGaP. We may have to think how some of the internal data
>>>> structures/SPIs should be updated to accomodate such requirements in future.
>>>>
>>>> *Orchestrator Component configurations*
>>>> I see alot of places where the orchestrator can have configurations. I
>>>> think its too early finalize them, but I think we can start refactoring
>>>> them out perhaps to the airavata-server.properties. I'm also seeing the
>>>> orchestrator is now hardcoded to use default/admin gateway and username.
I
>>>> think it should come from the request itself.
>>>>
>>>> *Visibility of API functions*
>>>> I think initialize(), shutdown() and startJobSubmitter() functions
>>>> should not be part of the API because I don't see a scenario where the
>>>> gateway developer would be responsible for using them. They serve a more
>>>> internal purpose of managing the orchestrator component IMO. As Amila
>>>> pointed out so long ago (wink) functions that do not concern outside
>>>> parties should not be used as part of the API.
>>>>
>>>> *Return values of Orchestrator API*
>>>> IMO unless it is specifically required to do so I think the functions
>>>> does not necessarily need to return anything other than throw exceptions
>>>> when needed. For example the launchExperiment can simply return void if all
>>>> is succesful and return an exception if something fails. Handling issues
>>>> with a try catch is not only simpler but also the explanations are readily
>>>> available for the user.
>>>>
>>>> *Data persisted in registry*
>>>> ExperimentRequest.getUsername() : I think we should clarify what this
>>>> username denotes. In current API, in experiment submission we consider two
>>>> types of users. Submission user (the user who submits the experiment to the
>>>> Airavata Server - this is inferred by the request itself) and the execution
>>>> user (the user who corelates to the application executions of the gateway
-
>>>> thus this user can be a different user for different gateway, eg: community
>>>> user, gateway user).
>>>> I think we should persist the date/time of the experiment request as
>>>> well.
>>>> Also when retrying of API functions in the case of a failure in an
>>>> previous attempt there should be a way to not to repeat already performed
>>>> steps or gracefully roleback and redo those required steps as necessary.
>>>> While such actions could be transparent to the user sometimes it might make
>>>> sense to allow user to be notified of success/failure of a retry. However
>>>> this might mean keeping additional records at the registry level.
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>>
>>>
>>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>

Mime
View raw message