airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <>
Subject Re: Orchestration Component implementation review
Date Mon, 20 Jan 2014 16:06:23 GMT
Thanks, Sachith.  Can you explain your question about APIs and SPIs a
little more? 


On 1/20/14 10:53 AM, Sachith Withana wrote:
> Hi All,
> I will go ahead and create the Wiki on the Orchestrator. Will send you all
> a draft as soon as I can.
> One question though, Do we have to explicitly show the SPIs and APIs both?
> On Mon, Jan 20, 2014 at 9:46 AM, Marlon Pierce <> wrote:
>> +1 for real use cases first. We have at least 3.  But I'm sure we will
>> want to make it as easy as possible for developers to pass back the
>> correct, created experimentID when invoking launchExperiment.
>> Marlon
>> On 1/17/14 2:57 PM, Saminda Wijeratne wrote:
>>> Marlon, I think until we put this to real use we wont get much feedback
>> on
>>> what aspects we should focus on more and in what features we should
>> expand
>>> or prioritize on. So how about having a test plan for the Orchestrator.
>>> Expose it to real usecases and see how it will survive. WDYT?
>>> It might be a little confusing to return a "JobRequest" object from the
>>> Orchestrator (since its a response). Or perhaps it should be renamed?
>>> Sachith, I think we should have a google hangout or a separate mail
>> thread
>>> (or both) to discuss muti-threaded support. Could you organize this
>> please?
>>> On Fri, Jan 17, 2014 at 10:29 AM, Amila Jayasekara
>>> <>wrote:
>>>> On Fri, Jan 17, 2014 at 10:32 AM, Saminda Wijeratne <
>>> wrote:
>>>>> Following are few thoughts I had during my review of the component,
>>>>> *Multi-threaded vs single threaded*
>>>>> If we are going to have multi-threaded job submission the
>> implementation
>>>>> should work on handling race conditions. Essentially JobSubmitter
>> should be
>>>>> able to "lock" an experiment request before continuing processing that
>>>>> request so that other JobSubmitters accessing the experiment requests
>> a the
>>>>> same time would skip it.
>>>> +1. These are implementation details.
>>>>> *Orchestrator service*
>>>>> We might want to think of the possibility in future where we will be
>>>>> having multiple deployments of an Airavata service. This could
>> particularly
>>>>> be true for SciGaP. We may have to think how some of the internal data
>>>>> structures/SPIs should be updated to accomodate such requirements in
>> future.
>>>> +1.
>>>>> *Orchestrator Component configurations*
>>>>> I see alot of places where the orchestrator can have configurations.
>>>>> think its too early finalize them, but I think we can start refactoring
>>>>> them out perhaps to the I'm also seeing the
>>>>> orchestrator is now hardcoded to use default/admin gateway and
>> username. I
>>>>> think it should come from the request itself.
>>>> +1. But in overall we may need to change the way we handle
>> configurations
>>>> within Airavata. Currently we have multiple configuration files and
>>>> multiple places where we read configurations. IMO we should have a
>> separate
>>>> module to handle configurations. Only this module should be aware how to
>>>> intepret configurations in the file and provide a component interface to
>>>> access those configuration values.
>>> +1 we tried this once with "ServerSettings" and "ApplicationSettings",
>> but
>>> apparently again more configuration files seems to have spawned. So far
>>> however they seemed to be localized for their component now.
>>>>> *Visibility of API functions*
>>>>> I think initialize(), shutdown() and startJobSubmitter() functions
>> should
>>>>> not be part of the API because I don't see a scenario where the gateway
>>>>> developer would be responsible for using them. They serve a more
>> internal
>>>>> purpose of managing the orchestrator component IMO. As Amila pointed
>> out so
>>>>> long ago (wink) functions that do not concern outside parties should
>> not be
>>>>> used as part of the API.
>>>> +1
>>>>> *Return values of Orchestrator API*
>>>>> IMO unless it is specifically required to do so I think the functions
>>>>> does not necessarily need to return anything other than throw
>> exceptions
>>>>> when needed. For example the launchExperiment can simply return void
>> if all
>>>>> is succesful and return an exception if something fails. Handling
>> issues
>>>>> with a try catch is not only simpler but also the explanations are
>> readily
>>>>> available for the user.
>>>> +1. Also try to have different exception for different scenarios. For
>>>> example if persistence (hypothetical) fails,
>> DatabasePersistenceException,
>>>> if validation fails, ValidationFailedException etc ... Then the
>> developer
>>>> who uses the API can catch these different exceptions and act on them
>>>> appropriately.
>>> +1. What needs to be understood here is that the Exception should be a
>>> Gateway friendly exception. i.e. it should not expose internal details of
>>> Airavata at the top-level exception and exception message should be self
>>> explanatory enough for the gateway developer not to remain scratching
>>> his/her head after reading the exception. A feedback from Sudhakar
>> sometime
>>> back was to provide suggestions in the exception message on how to
>> resolve
>>> the issue.
>>>>> *Data persisted in registry*
>>>>> ExperimentRequest.getUsername() : I think we should clarify what this
>>>>> username denotes. In current API, in experiment submission we consider
>> two
>>>>> types of users. Submission user (the user who submits the experiment
>> to the
>>>>> Airavata Server - this is inferred by the request itself) and the
>> execution
>>>>> user (the user who corelates to the application executions of the
>> gateway -
>>>>> thus this user can be a different user for different gateway, eg:
>> community
>>>>> user, gateway user).
>>>>> I think we should persist the date/time of the experiment request as
>>>>> well.
>>>> +1
>>>>>  Also when retrying of API functions in the case of a failure in an
>>>>> previous attempt there should be a way to not to repeat already
>> performed
>>>>> steps or gracefully roleback and redo those required steps as
>> necessary.
>>>>> While such actions could be transparent to the user sometimes it might
>> make
>>>>> sense to allow user to be notified of success/failure of a retry.
>> However
>>>>> this might mean keeping additional records at the registry level.
>>>> In addition we should also have a way of cleaning up unsubmitted
>>>> experiment ids. (But not sure whether you want to address this right
>> now).
>>>> The way I see this is to have a periodic thread which goes through the
>>>> table and clear up experiments which are not submitted for a defined
>> time.
>>> +1. Something else we may have to think of later is the data archiving
>>> capabilities. We keep running in to performance issues when the database
>>> grows with experiment results. Unless we become experts of distributed
>>> database management we should have a way better way to manage our db
>>> performance issues.
>>>> BTW, nice review notes, Saminda.
>>>> Thanks
>>>> Amila

View raw message