airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: Airavata 0.16 maven module organization
Date Tue, 02 Jun 2015 17:33:15 GMT
​Hi Suresh,

On Tue, Jun 2, 2015 at 12:15 PM, Suresh Marru <smarru@apache.org> wrote:

> Hi Shameera,
>
> Can you be concrete on what the problem you foresee and lets address it.
> If the problem is number if maven modules, is it really a problem?
>

​Problem is current maven dependency graph is complex and prone to have
cyclic maven dependency. One reason for this is, both implementations and
interfaces are reside in one maven module. It is clear that we need to
separate it out. Another reason is having large number of maven module
which also make dependency graph complex. For this we need to reduce number
of maven modules. And only have most required modules. So can you think
only this problem and explain how your suggestion resolve this problem?



>
> I certainly not explaining architecture through maven. What I am trying to
> address for 1.0 with this is:
>
> * Lets say we have validation implemented by component A version 1.3. We
> have new validation implementation which confirms to AbstractOrchestrator then
> it becomes Component A version 1.4 (which will have two implementations and
> have some switching logic at the factory class level). Instead, we might
> also have a totally new component G which will have its own internal
> interface and implementation and released as Component G version 1 and so
> on.
>
> * Lets say we want to add a new job submission to NERSC resources. We
> could add a new host and proceed. Alternately we wanted to try out
> integration with NEWT (https://newt.nersc.gov/). Instead of dovetailing
> the implementation to implement GFac interface, we then want to avoid
> adding BetterGFac or EvenBetterGfac interfaces to common core. We instead
> simply can develop a new component which implements submit job component
> interface. Internally it may have any implementations (programmed through
> Java interfaces and classes or scala or python or whatever).
>

Could you please explain why this is not possible with suggested maven
module refactoring?​

​Thanks,
Shameera.​



>
> So this organization is to facilitate components can evolve liberally
> without touching anything existing. In operational deployment we just
> figure out routing so at work queue level new component pick up jobs. At
> higher level, we can have the logic that if an experimental new component
> fails,it can put the job in the queue for  a proven stable component and so
> on.
>
> The down side is we add maven modules in order of number of competent
> multiplied by 2 or 3. On a up side, these components are totally loosely
> coupled and development evolves rapidly and provides deployment
> flexibility.
>
> Suresh
>
> On Jun 2, 2015, at 11:51 AM, Shameera Rathnayaka <shameerainfo@gmail.com>
> wrote:
>
> Hi Suresh,
>
> After reading this mail, what i understood is, you are not going to solve
> the problem we have, instead you are replacing one problem with another.
> Why we need maven module refactoring in the first place is to reduce
> unnecessary maven modules. What you are suggesting is replacing another set
> of modules with existing one. Instead explaining our architecture using
> maven modules we should use maven modules to manage project. let's use wiki
> to explain our architectural design. According to above structures how many
> maven modules will be there?
>
> Instead of planing 2.0 or any other release goals lets live with what we
> have now, let's talk about release 1.0 and then focus on 2.0 goals. For the
> 1.0 release do we need this level of categorization?
>
> Thanks,
> Shameera.
>
> On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <smarru@apache.org> wrote:
>
>> Hi All,
>>
>> While we let other thread on leaf modules going, here is a abstract
>> suggestion for maven modules. Please critique:
>>
>> We need to ensure the directory structure used for code navigability
>> should not created additional maven release artifacts. In 0.15 and earlier
>> versions, we have been releasing these empty artifacts for no good reason.
>>
>> ├── airavata-api
>> │   ├── airavata-api-interface-descriptions  (these are the thrift IDL’s)
>> │   ├── airavata-data-models (thrift generated data models, used by
>> clients, components and api server)
>> │   ├── airavata-api-java-stubs  (thrift generated java stubs used by
>> clients as well has servers for implementations)
>> │   ├── airavata-api-server (api service handlers and startup classes)
>> │   ├── api-security-manager (gsoc project to add API level oauth
>> security)
>> ├── clients   (self explanatory client SDK’s - thrift generated code plus
>> samples)
>> │   ├── airavata-client-cpp-sdk
>> │   ├── airavata-client-java-sdk
>> │   ├── airavata-client-php-sdk
>> │   ├── airavata-client-python-sdk
>> │   ├── airavata-sample-examples
>> │   └── airavata-xbaya-gui
>> ├── distribution
>> ├── integration-tests
>>
>> For component organization we can go any number of ways. The fundamental
>> proposition I have is components are literally loosely coupled and live by
>> themselves.
>>
>> One suggestion is to organize all component interface descriptions,
>> services and clients into and then component implementations (which
>> internally can have interfaces, implementations of these interfaces, util
>> packages and so forth).
>>
>> ├── components
>> │   ├── commons
>> │   ├── component-interface-descriptions
>> │   ├── component-services
>> │   │   ├── credential-store-service
>> │   │   ├── orchestrator-service
>> │   │   ├── task-executor-service
>> │   │   └── workflow-interpreter-service
>> │   ├── component-clients
>> │   │   ├── credential-store-client
>> │   │   ├── orchestrator-client
>> │   │   ├── task-executor-client
>> │   │   ├── workflow-interpreter-client
>> │   │   └── messaging
>> │   ├── orchestrator-component-impl
>> │   │   ├── orchestrator-interface
>> │   │   └── orchestrator-impl….
>> │   ├── registry
>> │   │   ├── app-catalog
>> │   │   ├── experiment-catalog
>> │   │   └── resource-catalog
>>
>> Other suggestion is to organize by a component. Within the component it
>> can also have the service interfaces and clients.
>>
>> This facilitates to have one more then one component implementing the
>> same component interface. They are used based on usecases and stability. In
>> no scenario we should have one component implementation depending on other
>> component’s implementation. If there is really a common functionality we
>> should have a module in commons used by both. At most the component should
>> only have dependency to component's thrift clients or in case of registry
>> and messaging appropriate java clients.
>>
>> This is again abstract, once we debate on the down sides and address
>> them, we can make this concrete to cover it all.
>>
>> Cheers,
>> Suresh
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Mime
View raw message