fineract-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sander van der Heyden <sandervanderhey...@musoni.eu>
Subject Re: Microservices
Date Tue, 24 May 2016 09:54:23 GMT
Hi guys,

First of all I think it's great that this discussion is now happening on
the public mailing list. It will first of all generate a lot of good
reading material for people looking to catch up on it later.

To put it very blunt, I think the ultimate decision to move to a new model
has indeed been made a while back before even starting the conversion into
an apache project. And from my side that is been a very good decision, as
we've seen that the current codebase has grown very rapidly over the last
few years, but also has grown increasingly complex. Part of this is due to
very free interpretation of the current architecture by the various devs
over time (including myself ;-)), but another part is also the scaling part
of the system.

As ever change won't come without it's challenges, and ultimately every use
of the current Fineract platform can decide to switchover to the new
infrastructure at a time they choose. If you are really against the
architecture design, or have built a very large proprietary ecosystem which
is tightly coupled into the current codebase, then there is always the
option to stay on the current Fineract version and continue to develop it
on your own, obviously without the large community that's currently there.

>From our side I want to first see a decent proof of concept, that shows
that the battles we are currently having in scaling the platform for our
customers are properly addressed. If that works, we will definitely hop on
and start contributing to it to ensure that any part of functionality that
we currently use which is not covered sufficiently will be addressed. But
until that time I'll be eagerly awaiting (and reading along) with these
threads to see the various updates and documents come forward.

@Keith: Great to see your back to dipping your toes into the project again!
I've picked up on your question to raise some of these in a separate thread
so will do so in the coming week or so (when I get some time to properly
write them down).

Sander

On Tue, May 24, 2016 at 11:23 AM, Keith Woodlock <keithwoodlock@gmail.com>
wrote:

> Myrle,
>
> Sounds like Markus/you have been on this journey for a while already.
> Everything sounds great. Be very interested in seeing it come together.
>
> For that reason I think the sooner you are transparent about where the code
> is the better so the community can take a look also and help out where
> possible.
>
> regards,
> Keith.
>
>
>
> On Tue, May 24, 2016 at 9:07 AM, Myrle Krantz <mkrantz@mifos.org> wrote:
>
> > Hey Keith,
> >
> > I’ll start with the easy questions:
> >
> > * Have I done this before? Yes.  But no woman is an island.  I’m happy to
> > have qualified advice.
> >
> > * Where will Markus and I be saving our work? We’ll send you a pointer
> once
> > we feel it would be productive for others to look at it.  Until we have a
> > common understanding of the design and technologies sharing code would be
> > fruitless.
> >
> > I thought about your suggestion for a general approach.  I agree we
> should
> > start with a limited scope, and I like your component break down.  I
> would
> > refine that as follows:  Since it’s always been important that
> > Mifos/Fineract is multi-tenanted, I think the best place to start is
> with a
> > service for tenant allocation and service orchestration.  From there, a
> > service for user/permission provisioning would be the obvious next step.
> > Together these represent your service 1.  Splitting them is not a
> > fundamentally different design.  It is cleaner though, since the user
> > management would be within a tenant, and the only service that should be
> > “tenant-aware” is the one that does tenant allocation.  For your second
> > service, you suggest attacking loans next.  Savings is just as important
> as
> > loans, and fundamentally equivalent (just black numbers versus red ones).
> > So after tenant allocation and user creation were complete, the next
> > bare-bones service should be a ledger.  Savings and loans products would
> > then be part of a fourth service for describing product offerings.
> Again,
> > I’ve taken what you represented as one service and made two out of it.
> >
> > All the services except the tenant creation service should treat all data
> > addition/adjustment as a command, just like you originally designed it
> for
> > Mifos.  That approach has worked well and we should stick to it.  But
> > command processing should be asynchronous for *everything*.  High volumes
> > of write operations in *any* service should not throttle capacity for any
> > other service.  The tenant creation service is the one service that can’t
> > use the command pattern, because commands are serialized to a database
> the
> > tenant creation service provisions.
> >
> > These are the components I just described:
> >
> > A.) A command module for writing and handling commands which would be
> used
> > by all services.  This is a compile time dependency.
> >
> > B.) A tenant service which creates the databases needed for the other
> > services, plus the tables needed for command persistence.
> >
> > C.) A user service which creates the users in the system.
> >
> > D.) A ledger service which tracks payments.  Depends on the command
> module,
> > the tenant service, and the user service.
> >
> > E.) A products service which can be used to configure savings and loans
> > products.  Depends on all the other components I described.
> >
> > B & C = Keith’s 1.)
> >
> > D & E = Keith’s 2.)
> >
> >
> > As we move forward we'll probably break it down further than that.
> >
> > These services should be deployed in jetty or tomcat much as today’s
> Mifos
> > is.  We can use Eureka and Ribbon for service discovery if we are
> deploying
> > multiple instances of a given service.  I’ve also really liked what I’ve
> > seen of docker.  I would like to ultimately build a docker image for each
> > service, and make the build scripts for those docker images part of the
> > service projects.
> >
> > This leaves me with a question for a mentor: As far as I can tell the
> > current mode of operation at Apache is one repository to one product.  I
> > would prefer to work with one repository per service.  I believe that
> would
> > help programmers remain strict about division of labor between the
> > services, and think more carefully about interface breaking changes.  Is
> > there any reason a product can’t have multiple repositories?
> >
> > Lest I create a false impression, this is not my design alone.  Markus
> > created most of it.  I’ve been refining it together with him.
> >
> >
> > Greets from the Voreifel, Germany
> >
> > Myrle
> >
> >
> >
> >
> > *Myrle Krantz*
> > Solutions Architect
> > RɅĐɅЯ, The Mifos Initiative
> > mkrantz@mifos.org | Skype: mkrantz.mifos.org | http://mifos.org
> > <http://facebook.com/mifos>  <http://www.twitter.com/mifos>
> >
> >
> > On Mon, May 23, 2016 at 11:16 AM, Keith Woodlock <
> keithwoodlock@gmail.com>
> > wrote:
> >
> > > Mryle,
> > >
> > > >> Now please excuse me while I get back to just doing it. ; o)
> > >
> > > Ok sounds like decision made that yourself and Markus will be going
> ahead
> > > with this :)
> > >
> > > Can you let us know where you will be checking in the code so we in the
> > > community can follow along.
> > >
> > > Given your are going to go ahead with developing a proof of concept
> > > starting from a blank canvas it might be good to layout a scope and
> some
> > > goals to get to in a time-boxed period of time. I would suggest
> something
> > > like:
> > >
> > > 1) A single microservice implemented from API to Database including
> > > deployment related capabilities
> > >
> > > The emphasis on this would be to show off the tech approach/pattern
> > > involved in implementing a microservice over the domain of the
> > microservice
> > > so feel free to choose a simpler area
> > >
> > > This hopefully could be provided in a much shorter time-line (a month
> or
> > > so)
> > >
> > > 2) A microservice that encapsulates the full Loan Product / Loan area
> of
> > > Fineract
> > >
> > > Makes sense to takcle the meat of the platform to show off qualities of
> > > microservices approach
> > >
> > > By the end of this you will have two microservice examples and that
> will
> > > bring up the questions around deployment and service discovery and how
> > that
> > > will be implemented.
> > >
> > > One last question would be to ask if Markus or yourself have actually
> > > developed/being part of a project previously that has used the
> > microservice
> > > approach specifically you are talking about?
> > >
> > > If yes thats great, if no, is it worth seeking someone with some
> > expertise
> > > in the area of microservices specifically in java area?
> > >
> > > regards,
> > > Keith.
> > >
> > >
> > > On Mon, May 23, 2016 at 8:26 AM, Myrle Krantz <mkrantz@mifos.org>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I won't be able to write a complete answer.  Nobody would read an
> > e-mail
> > > > that long.  So I’m going to “nibble” at it bit by bit.
> > > >
> > > > Of the three approaches Keith lists, I favor the third approach
> > > "Re-imagine
> > > > the fineract platform from ground up again (hope it solves the
> problems
> > > you
> > > > see and doesnt introduce new problems)".  I've been in the industry
> > long
> > > > enough (16 years) to understand that there are significant risks in
> > > > "starting over from scratch", and I hope that we can learn enough
> from
> > > the
> > > > existing code base to avoid some of those road bumps.
> > > >
> > > >
> > > > I would imagine multiple phases here:
> > > >
> > > > 1.) Markus and I start work on a framework, and basic services for
> this
> > > > refactoring.  Everyone else continues work on MifosX
> > > >
> > > > 2.) We do a proof of concept with a bare bones implementation.
> > > >
> > > > 3.) Report to the community based on what we learn by end of this
> year.
> > > >
> > > > 4.) Based on what we have learned, together with the community
> consider
> > > > migration of:
> > > >
> > > >    a.) New microservice-based code into the Fineract community.
> > > >
> > > >    b.) Community man-power from the existing codebase to the new code
> > > base.
> > > >
> > > >    c.) Existing customer data into the new microservice-based code
> via
> > > > migration scripts.
> > > >
> > > >    d.) Existing projects onto the new microservice-based product.
> > > >
> > > >    e.) The community app onto the new microservice-based backend.
> > > >
> > > > We can do a subset of the above migration steps.  There are several
> > > > possible approaches.  But I really want to do this together with the
> > > > Fineract community.  There’s much know-how here, and I think we can
> > copy
> > > > code over wholesale in many areas, and in other areas plagiarize good
> > > > architectural decisions and problem-solving approaches which the
> > existing
> > > > Fineract codebase illustrates.
> > > >
> > > > Because I want to do this with the Fineract community, I want to
> start
> > > > getting people in the community up to speed on the concepts early.
> > Even
> > > if
> > > > people decide they don't like this approach, nobody will be poorer
> for
> > > the
> > > > extra information.  I also very much want people’s feedback on where
> > they
> > > > see problems with our approach. I haven’t given much in the way of
> > > > specifics yet; I want people to have some background first. Which was
> > the
> > > > reason for my original mail.
> > > >
> > > > Now please excuse me while I get back to just doing it. ; o)
> > > >
> > > > (My next email will come tomorrow, and will focus on Roman’s very
> > > important
> > > > deployment question.)
> > > >
> > > > Greets from the Voreifel, Germany,
> > > > Myrle
> > > >
> > > >
> > > > *Myrle Krantz*
> > > > Solutions Architect
> > > > RɅĐɅЯ, The Mifos Initiative
> > > > mkrantz@mifos.org | Skype: mkrantz.mifos.org | http://mifos.org
> > > > <http://facebook.com/mifos>  <http://www.twitter.com/mifos>
> > > >
> > > >
> > > > On Thu, May 19, 2016 at 11:49 PM, Keith Woodlock <
> > > keithwoodlock@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Mryle,
> > > > >
> > > > > Thanks for your detailed reply, I added my comments (please
> remember
> > I
> > > am
> > > > > not involved in code base for sometime so might be missing alot)
> > below
> > > > but
> > > > > its a pretty long email so I summarise what I think here:
> > > > >
> > > > > The decision to move from one approach to developing a system to
> > > another
> > > > is
> > > > > a big one and not without its costs. I am not really for or against
> > it
> > > > and
> > > > > I'll be interested in seeing how things turn out whichever way the
> > > > > community decides to go. If everyone really feels it will be the
> > > solution
> > > > > to any problems that exist then they should go for it. The risk of
> > > course
> > > > > is that its a very tech solution to things.
> > > > >
> > > > > I think Chris Richardson does a good job on
> http://microservices.io
> > of
> > > > > outlining the differences between the monolith (
> > > > > http://microservices.io/patterns/monolithic.html) and the
> > > microservices
> > > > (
> > > > > http://microservices.io/patterns/microservices.html) approaches.
> > > > >
> > > > > You probably have the following 3 directions you can go:
> > > > >
> > > > > 1) Stay with the monolith and attempt to tackle the problems you
> see
> > > with
> > > > > the system (domain, performance etc)
> > > > >
> > > > > 2) Gradually move services from the monolith into microservices
> (hope
> > > it
> > > > > solves the problems you see and doesnt introduce new problems)
> > > > >
> > > > > 3) Re-imagine the fineract platform from ground up again (hope it
> > > solves
> > > > > the problems you see and doesnt introduce new problems)
> > > > >
> > > > > - Approach 2 sounds like it will be a drawn out affair and take a
> > long
> > > > time
> > > > > to reach the end with the drawback of having to support both
> > approaches
> > > > >
> > > > > - Approach 3 it too would be alot of effort but you could use
> > something
> > > > > like http://eventuate.io/ to get to a poof of concept faster
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Before devling into some of the points you raised I wanted to
> > highlight
> > > > two
> > > > > aspects of the fineract platform:
> > > > >
> > > > > *- Everything is exposed a HTTP API / Service*
> > > > >
> > > > > All the functionality that exists in the platform is or should be
> > > exposed
> > > > > by a HTTP API. This HTTP API is a techonology agonistic contract
> that
> > > > > developers can build their apps/clients on or integrate with
> easily.
> > > > >
> > > > > Developers can choose to build their apps/clients in any technology
> > > they
> > > > > wish and to use as much or as little of the platform API in their
> > app.
> > > > >
> > > > > API documented here https://demo.openmf.org/api-docs/apiLive.htm
> > > > >
> > > > > *- All HTTP API Endpoints deployed together*
> > > > >
> > > > > The code is compiled and bundled as a .war file that is deployed to
> > an
> > > > > application server like tomcat.
> > > > >
> > > > > If one application server isnt able to handle the request load you
> > can
> > > > > easily scale horizontally by adding more application servers (as
> the
> > > > > platform is stateless) and load balance the requests between the
> > > > serrvers.
> > > > > This would typically only be needed in a larger scale deployment.
> > > > >
> > > > >
> > > > > *The Problems You Listed*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *1.) We are starting to have performance problems.  Conflux in
> > > particular
> > > > > is running into this.  Mussoni is starting to have problems too.
> All
> > > of
> > > > us
> > > > > would like to move to bigger customers and bigger markets, so this
> > will
> > > > > only get worse.*
> > > > >
> > > > > You should open up a new email thread within the community on these
> > > > issues.
> > > > >
> > > > > It would be useful to provide some more details on what API
> requests
> > in
> > > > > praticular Conflux/Musoni and others are seeing performance
> problems
> > > > with.
> > > > >
> > > > >
> > > > > *2.) As we move to bigger customers, down-time becomes less and
> less
> > > > > acceptable.*
> > > > >
> > > > > There shouldnt need to be a need for downtime with the existing
> > > > > deploy-all-services-as-one approach.
> > > > >
> > > > > On tomcat you can deploy your new application in parallel with your
> > old
> > > > > one.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *3.) Loans products are hard-coded.  A lot of dev list mails read
> > like
> > > > this
> > > > > “For my case, I want my interest/fees calculated at a different
> time/
> > > on
> > > > a
> > > > > different basis than the ‘standard’ use case, how do I do this?”
> If
> > I
> > > > were
> > > > > a customer asking a question like this, some of the hacks being
> > > suggested
> > > > > would feel brittle and unsatisfying.  Savings products look like
> they
> > > > were
> > > > > implemented as an afterthought and are also hard-coded.*
> > > > >
> > > > > Sounds likes a domain modelling / requirements issue or an
> > > implementation
> > > > > issue.
> > > > >
> > > > > Different financial institutions in different countries have many
> > > > different
> > > > > flavours of the way they want their loan or saving products to work
> > and
> > > > its
> > > > > an area that will continue to change into the future.
> > > > >
> > > > > Mifos started out with supporting one or two of the common models
> in
> > > use
> > > > in
> > > > > the field and it looks like more have been added in since based on
> > > > content
> > > > > in http://fineract.incubator.apache.org/
> > > > >
> > > > > You might be suggesting though that its time to revisit the Loan
> > > Product
> > > > /
> > > > > Loan area and re-model it based on what you know today?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *4.) Our adherence to the CQRS pattern is less than perfect.
> Command
> > > > > responses were hard coded.  Some work has been done to make proper
> > > > command
> > > > > handlers out of it, but there are still semi-commands handled in a
> > big
> > > if
> > > > > else block.  Handling commands is synchronous even though one of
> the
> > > > > advantages of this pattern should be the ability to handle
> operations
> > > > > asynchronously.  This contributes to the problems mentioned in 1
> > > above.*
> > > > >
> > > > > Not sure what the hard-coded command response area is (been along
> > time
> > > > > since looked at code base)
> > > > >
> > > > > Its true that code that is responsible for identifying what command
> > > > handler
> > > > > to invoke based on the command is not great and suffered from the
> > 'god
> > > > > class' problems - the inital core of the platform was written in a
> > > > > time-boxed three month period as proof of concept and as more
> > > > apis/commands
> > > > > got added it got bigger.
> > > > >
> > > > > That said, if you have a preferred solution its surely an area that
> > > could
> > > > > of been improved in the last few years if it was that important?
> > > > >
> > > > > On the synchronous nature of them, yes we decided rather than going
> > for
> > > > an
> > > > > eventually-consistent system that the api calls would be like a
> > > > datatabase
> > > > > transaction call, i.e when you get a response you know its either
> > > > > successful or not
> > > > >
> > > > > Calling an api 1000's of times to do something in bulk of course
> > would
> > > > not
> > > > > make much sense and you would expect to add a bulk api for the that
> > > type
> > > > of
> > > > > activity.
> > > > >
> > > > > Also if there are very long running tasks then of course they
> should
> > > not
> > > > be
> > > > > done through a synchronous api call, you should set up an api that
> > > > supports
> > > > > 'long running tasks' and pass back a id/key that the user can use
> to
> > > > check
> > > > > for the result later or something similiar.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *5.) There’s currently no standard way to connect to what I like to
> > > call
> > > > > “money buses” (Swift, Stellar, UPI).  I did implement a bridge for
> > > > Stellar
> > > > > which listened for commands, but it wasn’t clear which commands to
> > > listen
> > > > > for.  I ended up deciding on journal entries.*
> > > > >
> > > > > Not fully across what you are doing here but if its a case of
> > > integrating
> > > > > some kind of third party payments api and invoking it based on some
> > api
> > > > > call like makePayment or disburse then that should be catered for
> by
> > > the
> > > > > platform. If you werent clear on how to do its probably cause the
> > > > platform
> > > > > didnt try to support that type of integration out of the box first
> of
> > > > all.
> > > > >
> > > > > Listening to journal entries might not be the best idea cause what
> > If i
> > > > > dont want to use the platforms accounting solution and turn it off?
> > > > >
> > > > > This use case should be designed into the praticular area in
> > question.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *6.) In first world core banking systems, the back end and the
> front
> > > end
> > > > > are carefully separated with clearly defined and limited
> > communication.
> > > > > This is to provide added protection against hacking of the type
> we’ve
> > > > seen
> > > > > in the news recently (Bangladesh’s central bank for example).*
> > > > >
> > > > > The front end (apps/clients) are totaly seperated from the back-end
> > api
> > > > at
> > > > > present. In terms of limiting what apps can speak to what APIs, you
> > > > > probably need to add that capability on top of the current
> > > > > authentication/authorization capablity.
> > > > >
> > > > > At present there is fine-grained support for restricting what apis
> a
> > > user
> > > > > is entitled to call, its not to much more to also limit what apis
> can
> > > be
> > > > > called based on what app/client is being used.
> > > > >
> > > > > *How these things related to microservices pionts*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *1.) By chopping up database accesses into logical units, we can
> > > increase
> > > > > capacity for only those units which need it.  This reduces costs.
> In
> > > > > particular, for the saving and reading of commands (for which we
> > expect
> > > > the
> > > > > largest volume of write instructions), we can switch to a faster
> > > > > persistence mechanism which works better in a distributed
> > environment.*
> > > > >
> > > > > Your probably talking about switching to storing the
> commands/events
> > > in a
> > > > > different persistence mechanism than a relational database. You
> could
> > > > > choose to do that now if you wish.
> > > > >
> > > > > You might also be referring to breaking the current database schema
> > > into
> > > > a
> > > > > database per service approach (
> > > > http://microservices.io/patterns/index.html
> > > > > ).
> > > > >
> > > > >
> > > > > I am not sure how this would reduce costs. Like any solution it has
> > its
> > > > > pros and cons (
> > > > > http://microservices.io/patterns/data/database-per-service.html)
> > > > >
> > > > >
> > > > >
> > > > > *2.) We can run multiple instances of a microservice, and, with
> > careful
> > > > > design, perform rolling updates, moving us towards the ability to
> > have
> > > > > 99.99% uptime.*
> > > > >
> > > > > Much like point 2 earlier,  you should be able to do this with the
> > > > current
> > > > > setup.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *3.) A more modular approach with well-defined APIs would make it
> > > > possible
> > > > > among other things to separate out the loans and savings products
> and
> > > let
> > > > > anyone implement their own loans and savings products who wants to.
> > > > Things
> > > > > like interest rates, fees, and repayments rates are very cultural,
> > and
> > > as
> > > > > we move into more countries we’ll need more flexibility here.*
> > > > >
> > > > > There should be well defined APIs at present. If you have a
> solution
> > > for
> > > > > allowing people to create their own loans and savings products with
> > > > > complete flexibiliy you should be able to roll it out now in the
> > > current
> > > > > setup.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *4.) CQRS is a truly awesome pattern for this domain, partly
> because
> > it
> > > > in
> > > > > theory, makes it possible to handle writes asynchronously.  If we
> can
> > > > > decouple the commands for different domains from each other and
> > > actually
> > > > > handle them asynchronously that would be a major step forward.  Of
> > > course
> > > > > this doesn’t necessitate a microservices deployment model, at least
> > not
> > > > > until the question of what we do with the data we create while
> > handling
> > > > the
> > > > > commands.  If, for example, a command for creating a user causes
> > > > different
> > > > > data to be adjusted then a command for starting a payment, why
> should
> > > > > creating a user cause a slowdown in working through payments?
> Those
> > > two
> > > > > kinds of processing don’t even have to execute on the same
> > hardware...*
> > > > >
> > > > > As you say, if you want to implement a more fully blown
> > implementation
> > > of
> > > > > the CQRS+Event Sourcing pattern you could implement that now for
> the
> > > > areas
> > > > > your believe need it and still respect the existing API
> > > > >
> > > > > The example you given isnt a good one as you wouldnt expect there
> to
> > > be a
> > > > > high throughput of users created but I understand that your saying
> > that
> > > > if
> > > > > there was 'intensive' operations going on on the platform, it might
> > be
> > > > > usefult to be able to split them into a seperate unit that runs on
> > > > seperate
> > > > > hardware so it doesnt impact more important functionality that
> might
> > > have
> > > > > an SLA on it like payment processing etc
> > > > >
> > > > > The question is, what are these intensive operations that might be
> > > > > affecting other areas of the platform so badly?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *5.) Writing ‘money bus’ plugins as microservices is a great way to
> > > > > separate functionality and be sure that even if one bus is down,
> the
> > > > others
> > > > > can still be up and running.  My stellar bridge component
> essentially
> > > > > applied this pattern.  If we can standardize the interface that
> > bridge
> > > > > components implement, we could make these easier to write and
> improve
> > > > Mifos
> > > > > connectivity to other systems.  These don’t need to be a standard
> > part
> > > of
> > > > > deployment, in different countries, different networks are popular,
> > and
> > > > > these things change over time too.  Connectivity to other systems
> is,
> > > in
> > > > my
> > > > > humble opinion, one of the most important factors in the success
> of a
> > > > > product.*
> > > > >
> > > > > Yes its very important to be able to integrate with other systems
> > and I
> > > > > dont see why the kind of functionality you want to implement and
> > > > integrate
> > > > > into mifos/fineract cant be done today in a clean way. I dont see
> how
> > > > > having all the current services split out into deployable units
> helps
> > > > this
> > > > > use case.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *6.) Separation of core systems from customer-visible systems will
> > > > require
> > > > > a duplication of some functionality.  This functionality would need
> > to
> > > > run
> > > > > in separate processes and only have access to data duplicated out
> of
> > > the
> > > > > back end system.  Most data would only flow from back to front
> > > (probably
> > > > > via some kind of ETL process).  If we divvy up our data based on
> the
> > > > > functionality groups which need it, it will be much easier to
> > determine
> > > > > which data should be copied “forward” and which should “stay
> behind”.
> > > A
> > > > > microservice is such a functionality group.*
> > > > >
> > > > > Its probably not a good idea to mix customer-visble functionality
> > with
> > > > core
> > > > > business-functionality. It terms of large financial institutions
> you
> > > > would
> > > > > be talking about 100,000s to millions of potentional users for
> > customer
> > > > > visible functionality whilst the core business-functionality would
> be
> > > > > significantly less.
> > > > >
> > > > > So they really are two different systems at some level.
> > > > >
> > > > > regards,
> > > > > Keith.
> > > > >
> > > > >
> > > > > On Thu, May 19, 2016 at 2:07 PM, Myrle Krantz <mkrantz@mifos.org>
> > > wrote:
> > > > >
> > > > > > This is a big topic, and I’m trying to decide if I should chop it
> > up
> > > > into
> > > > > > multiple threads.  I’m going to leave it as one for now, since I
> > > > detect a
> > > > > > healthy scepticism about the question of whether we need to do
> this
> > > at
> > > > > all,
> > > > > > and I don’t want to preclude that discussion.  Here are some of
> the
> > > > > > problems I’m seeing:
> > > > > >
> > > > > > 1.) We are starting to have performance problems.  Conflux in
> > > > particular
> > > > > is
> > > > > > running into this.  Mussoni is starting to have problems too.
> All
> > of
> > > > us
> > > > > > would like to move to bigger customers and bigger markets, so
> this
> > > will
> > > > > > only get worse.
> > > > > > 2.) As we move to bigger customers, down-time becomes less and
> less
> > > > > > acceptable.
> > > > > > 3.) Loans products are hard-coded.  A lot of dev list mails read
> > like
> > > > > this
> > > > > > “For my case, I want my interest/fees calculated at a different
> > time/
> > > > on
> > > > > a
> > > > > > different basis than the ‘standard’ use case, how do I do this?”
> > If
> > > I
> > > > > were
> > > > > > a customer asking a question like this, some of the hacks being
> > > > suggested
> > > > > > would feel brittle and unsatisfying.  Savings products look like
> > they
> > > > > were
> > > > > > implemented as an afterthought and are also hard-coded.
> > > > > > 4.) Our adherence to the CQRS pattern is less than perfect.
> > Command
> > > > > > responses were hard coded.  Some work has been done to make
> proper
> > > > > command
> > > > > > handlers out of it, but there are still semi-commands handled in
> a
> > > big
> > > > if
> > > > > > else block.  Handling commands is synchronous even though one of
> > the
> > > > > > advantages of this pattern should be the ability to handle
> > operations
> > > > > > asynchronously.  This contributes to the problems mentioned in 1
> > > above.
> > > > > > 5.) There’s currently no standard way to connect to what I like
> to
> > > call
> > > > > > “money buses” (Swift, Stellar, UPI).  I did implement a bridge
> for
> > > > > Stellar
> > > > > > which listened for commands, but it wasn’t clear which commands
> to
> > > > listen
> > > > > > for.  I ended up deciding on journal entries.
> > > > > > 6.) In first world core banking systems, the back end and the
> front
> > > end
> > > > > are
> > > > > > carefully separated with clearly defined and limited
> communication.
> > > > This
> > > > > > is to provide added protection against hacking of the type we’ve
> > seen
> > > > in
> > > > > > the news recently (Bangladesh’s central bank for example).
> > > > > >
> > > > > > So how does this relate to Microservices?  From here on out, I’m
> > > going
> > > > to
> > > > > > assume you all know what a Microservice is, so I hope you read
> > those
> > > > > links
> > > > > > in my original e-mail : o).
> > > > > >
> > > > > > 1.) By chopping up database accesses into logical units, we can
> > > > increase
> > > > > > capacity for only those units which need it.  This reduces costs.
> > In
> > > > > > particular, for the saving and reading of commands (for which we
> > > expect
> > > > > the
> > > > > > largest volume of write instructions), we can switch to a faster
> > > > > > persistence mechanism which works better in a distributed
> > > environment.
> > > > > > 2.) We can run multiple instances of a microservice, and, with
> > > careful
> > > > > > design, perform rolling updates, moving us towards the ability to
> > > have
> > > > > > 99.99% uptime.
> > > > > > 3.) A more modular approach with well-defined APIs would make it
> > > > possible
> > > > > > among other things to separate out the loans and savings products
> > and
> > > > let
> > > > > > anyone implement their own loans and savings products who wants
> to.
> > > > > Things
> > > > > > like interest rates, fees, and repayments rates are very
> cultural,
> > > and
> > > > as
> > > > > > we move into more countries we’ll need more flexibility here.
> > > > > > 4.) CQRS is a truly awesome pattern for this domain, partly
> because
> > > it
> > > > in
> > > > > > theory, makes it possible to handle writes asynchronously.  If we
> > can
> > > > > > decouple the commands for different domains from each other and
> > > > actually
> > > > > > handle them asynchronously that would be a major step forward.
> Of
> > > > course
> > > > > > this doesn’t necessitate a microservices deployment model, at
> least
> > > not
> > > > > > until the question of what we do with the data we create while
> > > handling
> > > > > the
> > > > > > commands.  If, for example, a command for creating a user causes
> > > > > different
> > > > > > data to be adjusted then a command for starting a payment, why
> > should
> > > > > > creating a user cause a slowdown in working through payments?
> > Those
> > > > two
> > > > > > kinds of processing don’t even have to execute on the same
> > > hardware...
> > > > > > 5.) Writing ‘money bus’ plugins as microservices is a great way
> to
> > > > > separate
> > > > > > functionality and be sure that even if one bus is down, the
> others
> > > can
> > > > > > still be up and running.  My stellar bridge component essentially
> > > > applied
> > > > > > this pattern.  If we can standardize the interface that bridge
> > > > components
> > > > > > implement, we could make these easier to write and improve Mifos
> > > > > > connectivity to other systems.  These don’t need to be a standard
> > > part
> > > > of
> > > > > > deployment, in different countries, different networks are
> popular,
> > > and
> > > > > > these things change over time too.  Connectivity to other systems
> > is,
> > > > in
> > > > > my
> > > > > > humble opinion, one of the most important factors in the success
> > of a
> > > > > > product.
> > > > > > 6.) Separation of core systems from customer-visible systems will
> > > > > require a
> > > > > > duplication of some functionality.  This functionality would need
> > to
> > > > run
> > > > > in
> > > > > > separate processes and only have access to data duplicated out of
> > the
> > > > > back
> > > > > > end system.  Most data would only flow from back to front
> (probably
> > > via
> > > > > > some kind of ETL process).  If we divvy up our data based on the
> > > > > > functionality groups which need it, it will be much easier to
> > > determine
> > > > > > which data should be copied “forward” and which should “stay
> > behind”.
> > > > A
> > > > > > microservice is such a functionality group.
> > > > > >
> > > > > > There are other advantages to microservices that are more general
> > in
> > > > > nature
> > > > > > and not specific to our use case; read the links I sent in my
> first
> > > > > e-mail
> > > > > > and you should start to get a feel for it.
> > > > > >
> > > > > > (Roman, I’m going to watch and see if your questions are answered
> > as
> > > we
> > > > > > discuss.  If not I’ll circle back to them, and see what was
> > missed.)
> > > > > >
> > > > > > Greets,
> > > > > > Myrle
> > > > > >
> > > > > >
> > > > > > *Myrle Krantz*
> > > > > > Solutions Architect
> > > > > > RɅĐɅЯ, The Mifos Initiative
> > > > > > mkrantz@mifos.org | Skype: mkrantz.mifos.org | http://mifos.org
> > > > > > <http://facebook.com/mifos>  <http://www.twitter.com/mifos>
> > > > > >
> > > > > >
> > > > > > On Thu, May 19, 2016 at 1:01 AM, Keith Woodlock <
> > > > keithwoodlock@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Myrle,
> > > > > > >
> > > > > > > First off, thanks for starting a conversation around this
> within
> > > the
> > > > > > > fineract dev community. I was wondering when this was going to
> > come
> > > > up.
> > > > > > >
> > > > > > > Rather than delving into the technicalities of what a
> > microservice
> > > is
> > > > > and
> > > > > > > isnt I thought it would be good if the people proposing this
> > could
> > > go
> > > > > > into
> > > > > > > a bit of detail here of why they are proposing it, the pros and
> > > cons
> > > > > of a
> > > > > > > microservice approach over the existing approach and the
> existing
> > > > > > problems
> > > > > > > within fineract/mifos that it will help solve/remove.
> > > > > > >
> > > > > > > I think that could promote some good discussion and following
> on
> > > from
> > > > > > that
> > > > > > > later we could delve into how it could be achieved if going
> that
> > > > > > direction.
> > > > > > >
> > > > > > > regards,
> > > > > > > Keith.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 18, 2016 at 11:14 PM, Roman Shaposhnik <
> > > > > roman@shaposhnik.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Myrle,
> > > > > > > >
> > > > > > > > microservices are indeed a nice architectural pattern with an
> > > > > > > exponentially
> > > > > > > > growing adoption in the enterprise. It typically leverages
> some
> > > > kind
> > > > > > of a
> > > > > > > > PaaS
> > > > > > > > solution to be available for deployment. My current
> > understanding
> > > > is
> > > > > > that
> > > > > > > > your
> > > > > > > > deployment model today is a traditional appserver. Are you
> > going
> > > to
> > > > > > > change
> > > > > > > > that?
> > > > > > > > If so, what will be the deployment requirements for the
> > project?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Roman.
> > > > > > > >
> > > > > > > > On Wed, May 18, 2016 at 2:35 AM, Myrle Krantz <
> > mkrantz@mifos.org
> > > >
> > > > > > wrote:
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > As many of you know Markus and I are thinking through a
> > > > > > rearchitecting
> > > > > > > of
> > > > > > > > > Fineract into microservices. Markus gave some of you a
> > preview
> > > > into
> > > > > > our
> > > > > > > > > work at the Mifos tech conference in Amsterdam in March
> > (those
> > > of
> > > > > you
> > > > > > > who
> > > > > > > > > are interested and weren’t able to attend can check it out
> on
> > > > > Youtube
> > > > > > > > here:
> > > > > > > > > https://www.youtube.com/watch?v=iIrnZpoNZ9A -- my
> apologies
> > > for
> > > > > the
> > > > > > > poor
> > > > > > > > > video quality)
> > > > > > > > >
> > > > > > > > > If you want to get an introductory understanding of what a
> > > > > > microservice
> > > > > > > > is,
> > > > > > > > > this is a good place to go:
> > > > > > > > >
> > > > > > > > > http://martinfowler.com/articles/microservices.html
> > > > > > > > >
> > > > > > > > > If there is interest, (and I have time), I’ll try to send
> you
> > > > more
> > > > > > > > > information about my ideas as they take shape.
> > > > > > > > >
> > > > > > > > > Greets,
> > > > > > > > >
> > > > > > > > > Myrle
> > > > > > > > >
> > > > > > > > > P.S.  Here’s Spring’s take on this:
> > > > > > > > >
> https://spring.io/blog/2015/07/14/microservices-with-spring
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Myrle Krantz*
> > > > > > > > > Solutions Architect
> > > > > > > > > RɅĐɅЯ, The Mifos Initiative
> > > > > > > > > mkrantz@mifos.org | Skype: mkrantz.mifos.org |
> > > http://mifos.org
> > > > > > > > > <http://facebook.com/mifos>  <http://www.twitter.com/mifos
> >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message