airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Airavata 0.16 maven module organization
Date Tue, 02 Jun 2015 17:58:18 GMT
On Jun 2, 2015, at 1:33 PM, Shameera Rathnayaka <shameerainfo@gmail.com> wrote:
> 
> ​Hi Suresh, 
> 
> On Tue, Jun 2, 2015 at 12:15 PM, Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
wrote:
> Hi Shameera,
> 
> Can you be concrete on what the problem you foresee and lets address it. If the problem
is number if maven modules, is it really a problem? 
> 
> ​Problem is current maven dependency graph is complex and prone to have cyclic maven
dependency. One reason for this is, both implementations and interfaces are reside in one
maven module. It is clear that we need to separate it out. 

This is a component level detail which needs to be handled within the components. We should
not unnecessary proliferate these into airavata level. The current cyclic dependencies may
be an oversight or lack of co-ordination at the component level. We just need to fix it. If
a component has too many implementations, it’s probably worth considering to spin it off
as a high level component. For instance, BES can be a component by itself implementing submitJob
thrift CPI instead of implementing GFac interface. If it is also using a GridFTP file transfer
module then it can be made into a library which both components use. 

A simple example is the recent effort in discarding EC2 implementations. If it was residing
in a component by itself and implementing a high level submitJob interface then it would have
prevailed. It is important to preserve such components as they can attract developers to fix
them. Currently we discard everything core developers do not support. This is not sustainable.


> Another reason is having large number of maven module which also make dependency graph
complex. For this we need to reduce number of maven modules. And only have most required modules.
So can you think only this problem and explain how your suggestion resolve this problem?

I do not have a good answer with a complex dependency graph. As long as it is navigable what
is the issue? is IDE not able to handle it? the module organization I am proposing is a higherarcial
and should make it manageable. If at all we think one repo should not have so many modules,
we should fork into multiple repositories (components can safely sit in separate repos by
themselves).  

> I certainly not explaining architecture through maven. What I am trying to address for
1.0 with this is:
> 
> * Lets say we have validation implemented by component A version 1.3. We have new validation
implementation which confirms to AbstractOrchestrator then it becomes Component A version
1.4 (which will have two implementations and have some switching logic at the factory class
level). Instead, we might also have a totally new component G which will have its own internal
interface and implementation and released as Component G version 1 and so on. 
> 
> * Lets say we want to add a new job submission to NERSC resources. We could add a new
host and proceed. Alternately we wanted to try out integration with NEWT (https://newt.nersc.gov/
<https://newt.nersc.gov/>). Instead of dovetailing the implementation to implement GFac
interface, we then want to avoid adding BetterGFac or EvenBetterGfac interfaces to common
core. We instead simply can develop a new component which implements submit job component
interface. Internally it may have any implementations (programmed through Java interfaces
and classes or scala or python or whatever). 
> 
> Could you please explain why this is not possible with suggested maven module refactoring?​


As I explained on earlier thread, it pulls together component level implementation details
(lower level java interfaces) into parent airavata core. This counters the conceptual architectural
patten that there is no common core. But Airavata constitutes loosely composed components
which are implementation agnostic to one another. 

Hope this explains all your concerns,
Suresh

> 
> ​Thanks, 
> Shameera.​
> 
>  
> 
> So this organization is to facilitate components can evolve liberally without touching
anything existing. In operational deployment we just figure out routing so at work queue level
new component pick up jobs. At higher level, we can have the logic that if an experimental
new component fails,it can put the job in the queue for  a proven stable component and so
on. 
> 
> The down side is we add maven modules in order of number of competent multiplied by 2
or 3. On a up side, these components are totally loosely coupled and development evolves rapidly
and provides deployment flexibility. 
> 
> Suresh
> 
>> On Jun 2, 2015, at 11:51 AM, Shameera Rathnayaka <shameerainfo@gmail.com <mailto:shameerainfo@gmail.com>>
wrote:
>> 
>> Hi Suresh, 
>> 
>> After reading this mail, what i understood is, you are not going to solve the problem
we have, instead you are replacing one problem with another. Why we need maven module refactoring
in the first place is to reduce unnecessary maven modules. What you are suggesting is replacing
another set of modules with existing one. Instead explaining our architecture using maven
modules we should use maven modules to manage project. let's use wiki to explain our architectural
design. According to above structures how many maven modules will be there?
>> 
>> Instead of planing 2.0 or any other release goals lets live with what we have now,
let's talk about release 1.0 and then focus on 2.0 goals. For the 1.0 release do we need this
level of categorization?
>> 
>> Thanks,
>> Shameera.
>> 
>> On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
wrote:
>> Hi All,
>> 
>> While we let other thread on leaf modules going, here is a abstract suggestion for
maven modules. Please critique:
>> 
>> We need to ensure the directory structure used for code navigability should not created
additional maven release artifacts. In 0.15 and earlier versions, we have been releasing these
empty artifacts for no good reason.
>> 
>> ├── airavata-api
>> │   ├── airavata-api-interface-descriptions  (these are the thrift IDL’s)
>> │   ├── airavata-data-models (thrift generated data models, used by clients,
components and api server)
>> │   ├── airavata-api-java-stubs  (thrift generated java stubs used by clients
as well has servers for implementations)
>> │   ├── airavata-api-server (api service handlers and startup classes)
>> │   ├── api-security-manager (gsoc project to add API level oauth security)
>> ├── clients   (self explanatory client SDK’s - thrift generated code plus
samples)
>> │   ├── airavata-client-cpp-sdk
>> │   ├── airavata-client-java-sdk
>> │   ├── airavata-client-php-sdk
>> │   ├── airavata-client-python-sdk
>> │   ├── airavata-sample-examples
>> │   └── airavata-xbaya-gui
>> ├── distribution
>> ├── integration-tests
>> 
>> For component organization we can go any number of ways. The fundamental proposition
I have is components are literally loosely coupled and live by themselves.
>> 
>> One suggestion is to organize all component interface descriptions, services and
clients into and then component implementations (which internally can have interfaces, implementations
of these interfaces, util packages and so forth).
>> 
>> ├── components
>> │   ├── commons
>> │   ├── component-interface-descriptions
>> │   ├── component-services
>> │   │   ├── credential-store-service
>> │   │   ├── orchestrator-service
>> │   │   ├── task-executor-service
>> │   │   └── workflow-interpreter-service
>> │   ├── component-clients
>> │   │   ├── credential-store-client
>> │   │   ├── orchestrator-client
>> │   │   ├── task-executor-client
>> │   │   ├── workflow-interpreter-client
>> │   │   └── messaging
>> │   ├── orchestrator-component-impl
>> │   │   ├── orchestrator-interface
>> │   │   └── orchestrator-impl….
>> │   ├── registry
>> │   │   ├── app-catalog
>> │   │   ├── experiment-catalog
>> │   │   └── resource-catalog
>> 
>> Other suggestion is to organize by a component. Within the component it can also
have the service interfaces and clients.
>> 
>> This facilitates to have one more then one component implementing the same component
interface. They are used based on usecases and stability. In no scenario we should have one
component implementation depending on other component’s implementation. If there is really
a common functionality we should have a module in commons used by both. At most the component
should only have dependency to component's thrift clients or in case of registry and messaging
appropriate java clients.
>> 
>> This is again abstract, once we debate on the down sides and address them, we can
make this concrete to cover it all.
>> 
>> Cheers,
>> Suresh
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> Best Regards,
>> Shameera Rathnayaka.
>> 
>> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com
<http://gmail.com/>
>> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>
> 
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com
<http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>

Mime
View raw message