airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Airavata 0.16 maven module organization
Date Tue, 02 Jun 2015 16:15:32 GMT
Hi Shameera,

Can you be concrete on what the problem you foresee and lets address it. If the problem is
number if maven modules, is it really a problem? 

I certainly not explaining architecture through maven. What I am trying to address for 1.0
with this is:

* Lets say we have validation implemented by component A version 1.3. We have new validation
implementation which confirms to AbstractOrchestrator then it becomes Component A version
1.4 (which will have two implementations and have some switching logic at the factory class
level). Instead, we might also have a totally new component G which will have its own internal
interface and implementation and released as Component G version 1 and so on. 

* Lets say we want to add a new job submission to NERSC resources. We could add a new host
and proceed. Alternately we wanted to try out integration with NEWT (https://newt.nersc.gov/
<https://newt.nersc.gov/>). Instead of dovetailing the implementation to implement GFac
interface, we then want to avoid adding BetterGFac or EvenBetterGfac interfaces to common
core. We instead simply can develop a new component which implements submit job component
interface. Internally it may have any implementations (programmed through Java interfaces
and classes or scala or python or whatever). 

So this organization is to facilitate components can evolve liberally without touching anything
existing. In operational deployment we just figure out routing so at work queue level new
component pick up jobs. At higher level, we can have the logic that if an experimental new
component fails,it can put the job in the queue for  a proven stable component and so on.


The down side is we add maven modules in order of number of competent multiplied by 2 or 3.
On a up side, these components are totally loosely coupled and development evolves rapidly
and provides deployment flexibility. 

Suresh

> On Jun 2, 2015, at 11:51 AM, Shameera Rathnayaka <shameerainfo@gmail.com> wrote:
> 
> Hi Suresh, 
> 
> After reading this mail, what i understood is, you are not going to solve the problem
we have, instead you are replacing one problem with another. Why we need maven module refactoring
in the first place is to reduce unnecessary maven modules. What you are suggesting is replacing
another set of modules with existing one. Instead explaining our architecture using maven
modules we should use maven modules to manage project. let's use wiki to explain our architectural
design. According to above structures how many maven modules will be there?
> 
> Instead of planing 2.0 or any other release goals lets live with what we have now, let's
talk about release 1.0 and then focus on 2.0 goals. For the 1.0 release do we need this level
of categorization?
> 
> Thanks,
> Shameera.
> 
> On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
wrote:
> Hi All,
> 
> While we let other thread on leaf modules going, here is a abstract suggestion for maven
modules. Please critique:
> 
> We need to ensure the directory structure used for code navigability should not created
additional maven release artifacts. In 0.15 and earlier versions, we have been releasing these
empty artifacts for no good reason.
> 
> ├── airavata-api
> │   ├── airavata-api-interface-descriptions  (these are the thrift IDL’s)
> │   ├── airavata-data-models (thrift generated data models, used by clients,
components and api server)
> │   ├── airavata-api-java-stubs  (thrift generated java stubs used by clients
as well has servers for implementations)
> │   ├── airavata-api-server (api service handlers and startup classes)
> │   ├── api-security-manager (gsoc project to add API level oauth security)
> ├── clients   (self explanatory client SDK’s - thrift generated code plus samples)
> │   ├── airavata-client-cpp-sdk
> │   ├── airavata-client-java-sdk
> │   ├── airavata-client-php-sdk
> │   ├── airavata-client-python-sdk
> │   ├── airavata-sample-examples
> │   └── airavata-xbaya-gui
> ├── distribution
> ├── integration-tests
> 
> For component organization we can go any number of ways. The fundamental proposition
I have is components are literally loosely coupled and live by themselves.
> 
> One suggestion is to organize all component interface descriptions, services and clients
into and then component implementations (which internally can have interfaces, implementations
of these interfaces, util packages and so forth).
> 
> ├── components
> │   ├── commons
> │   ├── component-interface-descriptions
> │   ├── component-services
> │   │   ├── credential-store-service
> │   │   ├── orchestrator-service
> │   │   ├── task-executor-service
> │   │   └── workflow-interpreter-service
> │   ├── component-clients
> │   │   ├── credential-store-client
> │   │   ├── orchestrator-client
> │   │   ├── task-executor-client
> │   │   ├── workflow-interpreter-client
> │   │   └── messaging
> │   ├── orchestrator-component-impl
> │   │   ├── orchestrator-interface
> │   │   └── orchestrator-impl….
> │   ├── registry
> │   │   ├── app-catalog
> │   │   ├── experiment-catalog
> │   │   └── resource-catalog
> 
> Other suggestion is to organize by a component. Within the component it can also have
the service interfaces and clients.
> 
> This facilitates to have one more then one component implementing the same component
interface. They are used based on usecases and stability. In no scenario we should have one
component implementation depending on other component’s implementation. If there is really
a common functionality we should have a module in commons used by both. At most the component
should only have dependency to component's thrift clients or in case of registry and messaging
appropriate java clients.
> 
> This is again abstract, once we debate on the down sides and address them, we can make
this concrete to cover it all.
> 
> Cheers,
> Suresh
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com
<http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>


Mime
View raw message