fineract-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Woodlock <keithwoodl...@gmail.com>
Subject Re: [jira] [Updated] (FINERACT-135) Performance improvement using EHCache or any other cache managers
Date Thu, 07 Apr 2016 12:24:20 GMT
Binny,

Are you are looking at total end to end time there of the API call? i.e
does that include latency of network? It looks like that might be the
backend log of the api transaction which would exclude any latency from
network. That kind of time seems crazy. Is this a production system you are
looking at or some test equivalent?

Dealing with just:

GET of loan:
{"startTime":1460024693065,"totalTime":16912,"method":"GET",
"url":"<<XXXX>>/mifosng-provider/api/v1/loans/295087","parameters":{"associations":["all"],"exclude":["guarantors"]}}

It would be worth delving deeper and showing all the SQL commands that are
executed agaisnt MYSQL when the GET Loan api is issued. 20 secs would make
anyone think you are missing an index or two in there but hard to know.

Do Musoni see the same performance for instance on any of their DBs?

Keith.


On Thu, Apr 7, 2016 at 12:53 PM, Binny Gopinath Sreevas <
binny.gopinath@gmail.com> wrote:

> Keith, Markus, Adi,
>
> Accounting is enabled. And it is synchronously tied to loan transactions. I
> am looking at Catalina.out. Hence, I don't think it could be network
> latency.
>
> Fetch of a loan takes about 20+ seconds  and about another 8-9 seconds for
> fetching the data tables associated to the loan.
> (Fyi, each loan has 2-3 fees, and about 52 entries in repayment schedule -
> and hence, potentially that many transactions and journal entries as the
> client makes repayments)
> The GET is being called with parameters
> "associations=all&exclude=guarantors".
>
> The collection sheet APIs -
> Generate collection sheet takes about 18 seconds
> The saving of the collection sheet takes about - . 60+ seconds
>
> Examples:
> GET of loan:
> {"startTime":1460024693065,"totalTime":16912,"method":"GET",
>
> "url":"<<XXXX>>/mifosng-provider/api/v1/loans/295087","parameters":{"associations":["all"],"exclude":["guarantors"]}}
>
> GET of data tables associated to Loans:
> {"startTime":1460024693441,"totalTime":9609,"method":"GET",
>
> "url":"<<XXXX>>/mifosng-provider/api/v1/datatables/PPI%20Details/295087","parameters":{"genericResultSet":["true"]}}
>
> POST of loans:
> {"startTime":1460024697085,"totalTime":17197,"method":"POST",
> "url":"<<XXXX>>/mifosng-provider/api/v1/loans","parameters":{}}
>
> Generate collection sheet:
> {"startTime":1460029687320,"totalTime":15564,"method":"POST",
>
> "url":"<<XXXX>>/mifosng-provider/api/v1/centers/50689","parameters":{"command":["generateCollectionSheet"]}}
>
> Save collection sheet:
> {"startTime":1460024733759,"totalTime":69372,"method":"POST",
>
> "url":"<<XXXX>>/mifosng-provider/api/v1/centers/64305","parameters":{"command":["saveCollectionSheet"]}}
>
> Note: This is still MIFOS code deployed and not yet upgraded to FINERACT -
> which will happen in couple of weeks. I am not expecting FINERACT
> performance to be mcuh different.
>
> The environment is: tomcat running on a Linux box with 16GB RAM and
> database on a separate box with 32GB RAM. Both of them on AWS.
>
> - Binny
>
>
>
> On Thu, Apr 7, 2016 at 3:58 PM, Adi Raju <adi.raju@confluxtechnologies.com
> >
> wrote:
>
> > As the loan portfolio size increases, performance of most of loan
> > transaction APIs is seen to be slow, more specifically collection sheet
> and
> > repayments.
> > In any loan related transaction, Loan entity object is created. Initial
> > observation is that creation of this Loan object is resulting in around
> > 40-50 select queries to DB.
> > Also there are so many other tables like office/user/GLaccounts/code
> values
> > etc which rarely change but again result in queries to DB.
> >
> > Hibernate 2nd Level cache and possibly JOIN FETCH strategies could help
> to
> > start with.
> >
> > Regards,
> > Adi
> >
> >
> > -----Original Message-----
> > From: Markus GeiƟ [mailto:markus.geiss@live.de]
> > Sent: 07 April 2016 15:46
> > To: dev@fineract.incubator.apache.org
> > Subject: RE: [jira] [Updated] (FINERACT-135) Performance improvement
> using
> > EHCache or any other cache managers
> >
> > Hey,
> > as far as I know EHCache is used to enable 2nd levelcache for Hibernate,
> so
> > a per node cache.
> > If we want upfront distributed caching I'd vote forHazelcast, because
> we'd
> > need to implement it anyways.
> > I back Keith' question, it would be good to know whatthe issue is to
> > understand the needed adjustments.
> > Is the bottleneck on the service side, is it DB related,or maybe the
> > network. Where we loose performance here?
> >
> > Best,
> >
> > Markus
> >
> > .::YAGNI likes a DRY KISS::.
> >
> > > Date: Thu, 7 Apr 2016 10:04:16 +0100
> > > Subject: Re: [jira] [Updated] (FINERACT-135) Performance improvement
> > > using EHCache or any other cache managers
> > > From: keithwoodlock@gmail.com
> > > To: dev@fineract.incubator.apache.org
> > >
> > > Binny,
> > >
> > > >> In a production environment wiith about 130,000 clients with loans
> > > >> - the
> > > performance of the loan APIs are poor.
> > >
> > > This statement is too vague. What functionality of loans is said to
> > > have poor performance. Is it the fetch part of loans of the
> > > transactions that happen over the loans like make payment etc which if
> > > I remember interact closely with the Accounting area. Is accounting
> > > turned on/configured when performance is poor? Is it 'synchronously'
> > > coupled to the loans functionality as part of loan transactions.
> > >
> > > Caches will only help on the GET/Fetch side of things.
> > >
> > > Is distributed caching supported on Mifos using EHcache or is it just
> > > single node cache? Some like Hazelcast or similiar might be a good
> > > option if wanting to support distributed caching.
> > >
> > > regards,
> > > Keith.
> > >
> > > On Thu, Apr 7, 2016 at 9:57 AM, Binny Gopinath Sreevas (JIRA) <
> > > jira@apache.org> wrote:
> > >
> > > >
> > > >      [
> > > > https://issues.apache.org/jira/browse/FINERACT-135?page=com.atlassia
> > > > n.jira.plugin.system.issuetabpanels:all-tabpanel
> > > > ]
> > > >
> > > > Binny Gopinath Sreevas updated FINERACT-135:
> > > > --------------------------------------------
> > > >     Description:
> > > > In a production environment wiith about 130,000 clients with loans -
> > > > the performance of the loan APIs are poor. I have been told is that
> > > > enabling EHCache and making it work will definitely improve the
> > > > performance. Can this be done and tested for deployment scenarios -
> > > > including multi-tenant setup and load balanced (multiple
> > > > tomcats/FINERACT instances which talk to the same database).
> > > >
> > > > Reference to previous work on this:
> > > >
> > > > https://mifosforge.jira.com/wiki/display/projects/GSOC+2013+-+Distri
> > > > buted+Caching+support
> > > >
> > > >   was:
> > > > In a production environment wiith about 130,000 clients with loans -
> > > > the performance of the loan APIs are poor. I have been told is that
> > > > enabling EHCache and making it work will definitely improve the
> > > > performance. Can this be done and tested for deployment scenarios -
> > > > including multi-tenant setup and load balanced (multiple
> > > > tomcats/FINERACT instances which talk to the same database).
> > > >
> > > >
> > > >
> > > >
> > > > > Performance improvement using EHCache or any other cache managers
> > > > > -----------------------------------------------------------------
> > > > >
> > > > >                 Key: FINERACT-135
> > > > >                 URL:
> > https://issues.apache.org/jira/browse/FINERACT-135
> > > > >             Project: Apache Fineract
> > > > >          Issue Type: Improvement
> > > > >            Reporter: Binny Gopinath Sreevas
> > > > >            Assignee: Markus Geiss
> > > > >            Priority: Critical
> > > > >
> > > > > In a production environment wiith about 130,000 clients with loans
> > > > > - the
> > > > performance of the loan APIs are poor. I have been told is that
> > > > enabling EHCache and making it work will definitely improve the
> > > > performance. Can this be done and tested for deployment scenarios -
> > > > including multi-tenant setup and load balanced (multiple
> > > > tomcats/FINERACT instances which talk to the same database).
> > > > > Reference to previous work on this:
> > > > >
> > > > https://mifosforge.jira.com/wiki/display/projects/GSOC+2013+-+Distri
> > > > buted+Caching+support
> > > >
> > > >
> > > >
> > > > --
> > > > This message was sent by Atlassian JIRA
> > > > (v6.3.4#6332)
> > > >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message