airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Too Many Leaf Modules.
Date Sun, 31 May 2015 02:45:25 GMT
Shameera,

Every component has in its own thrift service interface (registry and messaging have exceptions).
Every component will need to have a dependency to airavata data models (which includes util
classes) and probably registry and messaging. if a component A needs to invoke component B
via RPC call, then it just needs to include its component A’s thrift client. If the communication
is through work queue’s then there is no dependency between them. Can you describe what
you want to propose in this context? 

Suresh

> On May 30, 2015, at 8:36 PM, Shameera Rathnayaka <shameerainfo@gmail.com> wrote:
> 
> Hi Suresh, 
> 
> Spark is not the right comparison for this discussion. I have been a spark incubation
mentor and have been following the code organization since its early days. All of the spark
components you mentions rely on the core. 
> 
> ​Yes, that is what I highlight , each components doesn't have their own core modules
which has the interfaces. Spark SQL module has core submodule but all interfaces reside in
main core module.
> 
> 
> Let me step back and ask, what is the problem you are trying to solve? We sure need to
cleanup modules and it is time to re-look at the component organization. But what do you want
to really achieve by combining all the core into monolithic components. It took an effort
to cleanly separate functionality so they can evolve and can be improved independently. 
> 
> 
> I am not suggesting to go back to the monolithic core which has all implementations and
interfaces bundle with together. What i am saying is have core interfaces all together, this
will give us clear module dependency graph. This is something like having one root dependency
graph instead of multiple roots. As a developer it is cumbersome and hard to deal with module
dependency issues. Having too fine gran core modules introduce wrong dependency graph eventually,
and it will prevent us to follow proper design patterns in our code base. If we avoid design
patterns, it will require more time to find bugs and maintain. This is what I am trying to
resolve. I have first hand experiences with current airavata code. If you see the current
module dependency graphs, then you will understand why I am making such noise to resolve this.
> 
> 
>> So why do we make our repository bulky with modules unless it doesn't provide any
considerable advantage. 
>> ​If we really w​ant to ​separate distribution bundle for each component ( apiServer
, Orchestrator and Gfac ) let​'​s use different bin.xml file​s​ to do it instead of
using different modules. But reality is we only use ​all in one distribution.
> 
> Once the monitoring is fixed to use messaging, we really need to decouple the component
deployments. Yes there is a considerable advantage. Each components has different quality
of service requirements. A production platform has to load balance and scale horizontally.
And thats different for different component. The all in one bundle has 300+ jars, but API
server and orchestrator when independent will have around 50 or so jars. When I want to deploy
api server and orchestrator there is significant different in small light weight components
vs one monolithic core. 
> 
> Another problem is component evolution. Lets say there is a production deployment running
1.1.4 version. Lets say the single job execution is stable enough and there is a 6 month focused
effort on workflow. Say the master moves to 1.2.8 with all the changes to workflow and only
few to single application execution. We can more comfortably upgrade is they are cleanly separated
modules. But if it is one core and so many changes to it (even though technically they are
to different classes, the perception will remain), the upgrades will get behind.
> 
> Bottom line I am + 1 for cleaning up the modules. Past few years we have been moving
towards micro service architectures and your suggestions will reverse this back to monolithic
architecture. I am -1 for this change in direction. 
> 
> 
> By looking at your miniature component suggestion above, it has 30+ modules. Do you think
we really need this number of categorization? With my industrial experience, I have seen ​number
of modules in a project always increase with time. Hence​ If we start 30+ we will come to
40+ and then 50+​ and so on so forth​.  Why we make this complicated? 
> 
> ​Thanks,
> Shameera.​
> 
>  
> 
> Suresh
> 
>> 
>> ​​Thanks,
>> ​Shameera.
>> ​
>> 
>> Suresh
>> 
>>> On May 29, 2015, at 11:29 AM, Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
wrote:
>>> 
>>> + 1. 
>>> 
>>> I was planning to bring up this issue also. Probably it will not address what
you are raising, but here is a tree output from airavata labs code I was toying with locally.
I did not yet compare it with what you proposed, I will do so later today.
>>> 
>>> ├── airavata-api
>>> │   ├── airavata-api-interface-descriptions
>>> │   ├── airavata-api-java-stubs
>>> │   ├── airavata-api-server
>>> │   ├── airavata-data-models
>>> │   ├── api-security-manager
>>> ├── clients
>>> │   ├── airavata-client-cpp-sdk
>>> │   ├── airavata-client-java-sdk
>>> │   ├── airavata-client-php-sdk
>>> │   ├── airavata-client-python-sdk
>>> │   ├── airavata-sample-examples
>>> │   └── airavata-xbaya-gui
>>> ├── components
>>> │   ├── commons
>>> │   ├── component-interface-descriptions
>>> │   ├── component-services
>>> │   │   ├── credential-store-service
>>> │   │   ├── orchestrator-service
>>> │   │   ├── task-executor-service
>>> │   │   └── workflow-interpreter-service
>>> │   ├── component-clients
>>> │   │   ├── credential-store-client
>>> │   │   ├── orchestrator-client
>>> │   │   ├── task-executor-client
>>> │   │   ├── workflow-interpreter-client
>>> │   │   └── messaging
>>> │   ├── task-adaptors
>>> │   │   ├── compute
>>> │   │   └── data-movement
>>> │   ├── registry
>>> │   │   ├── app-catalog
>>> │   │   ├── experiment-catalog
>>> │   │   └── resource-catalog
>>> │   └── workflow-interpreter
>>> ├── distribution
>>> ├── integration-tests
>>> 
>>> 
>>> 
>>>> On May 29, 2015, at 10:15 AM, Shameera Rathnayaka <shameera@apache.org
<mailto:shameera@apache.org>> wrote:
>>>> 
>>>> Hi Devs, 
>>>> 
>>>> As we are using different modules to package different type of functionalities,
which will help us to maintain loosely couple codes. Now the project has 49 leaf module (
one to hit half century :) ). If we allow project to grow this way, having too fine grain
modules will be huge headache in future. IMO we should clean this ASAP before it become really
mess. Actually we half way there, I experienced cyclic dependency issues when I was writing
workflow implementation and email monitoring. Please see the modules in current repo below.

>>>> 
>>>> <module-name> ( <num of child modules> )
>>>> 
>>>> modules  ( 43 )
>>>>      app-catalog ( 2 )
>>>>      commons ( 1 )
>>>>      configurations ( 2 )
>>>>      credential-store ( 3 )
>>>>      distribution ( 8 )
>>>>      gfac ( 10 )
>>>>      integration test ( 1 )
>>>>      messaging ( 2 )
>>>>      orchestrator ( 3 )
>>>>      registry ( 3 )
>>>>      security ( 1 )
>>>>      server ( 1 )
>>>>      test-suit ( 1 )
>>>>      workflow ( 1 )
>>>>      workflow-modal ( 3 )
>>>>      xbaya ( 1 ) 
>>>> airavata-api ( 5 )
>>>> tools ( 1 ) 
>>>> 
>>>> Most of the current modules have interfaces and implementations together,
but this violate our main goal which reduce inter module dependencies. Following is what I
am suggesting, WDYS?
>>>> 
>>>> core { has all core interfaces and basic classes of gfac-core , orchestrator-core
, message-core , monitor core, registry core, workflow-core}
>>>> service - all thrift services and service handlers 
>>>> orchestrator - orchestrator specific classes
>>>> gfac 
>>>>      SSH  
>>>>      BES
>>>>      Local
>>>> message - amqp implemention 
>>>> distribution 
>>>>      XBaya
>>>>      server - { use different mode input to start server as orchestrator
, Gfac or/and api-server }
>>>> commons
>>>> registry
>>>> app-catalog
>>>> security
>>>> Workflow
>>>> XBaya-gui
>>>> Integration-test 
>>>> 
>>>> Thanks, 
>>>> Shameera.
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> Best Regards,
>> Shameera Rathnayaka.
>> 
>> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com
<http://gmail.com/>
>> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com
<http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>

Mime
View raw message