airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Too Many Leaf Modules.
Date Sat, 30 May 2015 22:57:09 GMT
On May 30, 2015, at 12:34 PM, Shameera Rathnayaka <shameerainfo@gmail.com> wrote:
> 
> Hi Suresh, 
> 
> I do not think bundling all core into one module is a good idea. When there is distinct
enough functionality why combine? There are advantages with micro-service architecture both
in terms of rapid evolution as well as ease of light weight deployments. For instance lets
take API server and Orchestrator example. API server gets hit for all the calls, but only
a fraction of them are intended for Orchestrator. Even in your approach of starting different
services from one module, the dependencies are all class loaded (unless we put sophistication
in startup scripts).  
> 
> If two modules have significant overlapping functionality then + 1 to discuss those as
case by case and combine them as necessary. But can you explain a little more on why you want
to combine distinct functional modules? 
> 
> ​A good example is Apache Spark, it is a rapid evolving project, and it also has different
major components ( Sprak SQL, Spark Streaming, MLlib and GraphX) ​ ​but all share one
core component[1]. No doubt they need light weight deployments more than us, as they have
higher user community.

Spark is not the right comparison for this discussion. I have been a spark incubation mentor
and have been following the code organization since its early days. All of the spark components
you mentions rely on the core. 

Let me step back and ask, what is the problem you are trying to solve? We sure need to cleanup
modules and it is time to re-look at the component organization. But what do you want to really
achieve by combining all the core into monolithic components. It took an effort to cleanly
separate functionality so they can evolve and can be improved independently. 

> So why do we make our repository bulky with modules unless it doesn't provide any considerable
advantage. 
> ​If we really w​ant to ​separate distribution bundle for each component ( apiServer
, Orchestrator and Gfac ) let​'​s use different bin.xml file​s​ to do it instead of
using different modules. But reality is we only use ​all in one distribution.

Once the monitoring is fixed to use messaging, we really need to decouple the component deployments.
Yes there is a considerable advantage. Each components has different quality of service requirements.
A production platform has to load balance and scale horizontally. And thats different for
different component. The all in one bundle has 300+ jars, but API server and orchestrator
when independent will have around 50 or so jars. When I want to deploy api server and orchestrator
there is significant different in small light weight components vs one monolithic core. 

Another problem is component evolution. Lets say there is a production deployment running
1.1.4 version. Lets say the single job execution is stable enough and there is a 6 month focused
effort on workflow. Say the master moves to 1.2.8 with all the changes to workflow and only
few to single application execution. We can more comfortably upgrade is they are cleanly separated
modules. But if it is one core and so many changes to it (even though technically they are
to different classes, the perception will remain), the upgrades will get behind.

Bottom line I am + 1 for cleaning up the modules. Past few years we have been moving towards
micro service architectures and your suggestions will reverse this back to monolithic architecture.
I am -1 for this change in direction. 

Suresh

> 
> ​​Thanks,
> ​Shameera.
> ​
> 
> Suresh
> 
>> On May 29, 2015, at 11:29 AM, Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
wrote:
>> 
>> + 1. 
>> 
>> I was planning to bring up this issue also. Probably it will not address what you
are raising, but here is a tree output from airavata labs code I was toying with locally.
I did not yet compare it with what you proposed, I will do so later today.
>> 
>> ├── airavata-api
>> │   ├── airavata-api-interface-descriptions
>> │   ├── airavata-api-java-stubs
>> │   ├── airavata-api-server
>> │   ├── airavata-data-models
>> │   ├── api-security-manager
>> ├── clients
>> │   ├── airavata-client-cpp-sdk
>> │   ├── airavata-client-java-sdk
>> │   ├── airavata-client-php-sdk
>> │   ├── airavata-client-python-sdk
>> │   ├── airavata-sample-examples
>> │   └── airavata-xbaya-gui
>> ├── components
>> │   ├── commons
>> │   ├── component-interface-descriptions
>> │   ├── component-services
>> │   │   ├── credential-store-service
>> │   │   ├── orchestrator-service
>> │   │   ├── task-executor-service
>> │   │   └── workflow-interpreter-service
>> │   ├── component-clients
>> │   │   ├── credential-store-client
>> │   │   ├── orchestrator-client
>> │   │   ├── task-executor-client
>> │   │   ├── workflow-interpreter-client
>> │   │   └── messaging
>> │   ├── task-adaptors
>> │   │   ├── compute
>> │   │   └── data-movement
>> │   ├── registry
>> │   │   ├── app-catalog
>> │   │   ├── experiment-catalog
>> │   │   └── resource-catalog
>> │   └── workflow-interpreter
>> ├── distribution
>> ├── integration-tests
>> 
>> 
>> 
>>> On May 29, 2015, at 10:15 AM, Shameera Rathnayaka <shameera@apache.org <mailto:shameera@apache.org>>
wrote:
>>> 
>>> Hi Devs, 
>>> 
>>> As we are using different modules to package different type of functionalities,
which will help us to maintain loosely couple codes. Now the project has 49 leaf module (
one to hit half century :) ). If we allow project to grow this way, having too fine grain
modules will be huge headache in future. IMO we should clean this ASAP before it become really
mess. Actually we half way there, I experienced cyclic dependency issues when I was writing
workflow implementation and email monitoring. Please see the modules in current repo below.

>>> 
>>> <module-name> ( <num of child modules> )
>>> 
>>> modules  ( 43 )
>>>      app-catalog ( 2 )
>>>      commons ( 1 )
>>>      configurations ( 2 )
>>>      credential-store ( 3 )
>>>      distribution ( 8 )
>>>      gfac ( 10 )
>>>      integration test ( 1 )
>>>      messaging ( 2 )
>>>      orchestrator ( 3 )
>>>      registry ( 3 )
>>>      security ( 1 )
>>>      server ( 1 )
>>>      test-suit ( 1 )
>>>      workflow ( 1 )
>>>      workflow-modal ( 3 )
>>>      xbaya ( 1 ) 
>>> airavata-api ( 5 )
>>> tools ( 1 ) 
>>> 
>>> Most of the current modules have interfaces and implementations together, but
this violate our main goal which reduce inter module dependencies. Following is what I am
suggesting, WDYS?
>>> 
>>> core { has all core interfaces and basic classes of gfac-core , orchestrator-core
, message-core , monitor core, registry core, workflow-core}
>>> service - all thrift services and service handlers 
>>> orchestrator - orchestrator specific classes
>>> gfac 
>>>      SSH  
>>>      BES
>>>      Local
>>> message - amqp implemention 
>>> distribution 
>>>      XBaya
>>>      server - { use different mode input to start server as orchestrator , Gfac
or/and api-server }
>>> commons
>>> registry
>>> app-catalog
>>> security
>>> Workflow
>>> XBaya-gui
>>> Integration-test 
>>> 
>>> Thanks, 
>>> Shameera.
>> 
> 
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com
<http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>

Mime
View raw message