Mailing-List: contact dev-help@fineract.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@fineract.incubator.apache.org
Date: Tue, 18 Apr 2017 06:38:41 +0000 (UTC)
From: "Avik Ganguly (JIRA)" <jira@apache.org>
To: dev@fineract.incubator.apache.org
Message-ID: <JIRA.13064591.1492497359000.302993.1492497521579@Atlassian.JIRA>
In-Reply-To: <JIRA.13064591.1492497359000@Atlassian.JIRA>
References: <JIRA.13064591.1492497359000@Atlassian.JIRA> <JIRA.13064591.1492497359839@jira-lw-us.apache.org>
Subject: [jira] [Updated] (FINERACT-428) Parallelization of Jobs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 18 Apr 2017 06:38:47 -0000


     [ https://issues.apache.org/jira/browse/FINERACT-428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Avik Ganguly updated FINERACT-428:
----------------------------------
    Description: 
For starters, it will be useful to have some technical configuration added as job parameter for each of the below jobs, that is batch size and thread pool size for executor service so this has a dependency on FINERACT-425. 

Add migration script to add those 2 configurations to each of the following jobs with the value of batch size being 500 and thread pool size being 16 :-

Add Accrual Transactions
Add Periodic Accrual Transactions
Add Accrual Transactions For Loans With Income Posted As Transactions
Generate Loan Loss Provisioning
Post Interest for Savings
Update Loan Summary

This would require separation of core functionality to a separate class.

Simplified example :-

```
final ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
        
        final LocalDate dueDate = DateUtils.getLocalDateOfTenant();
        final Collection<LoanAccountData> loansToBeRepaidData = this.loanReadPlatformService
                .retrieveLoansToBeRepaidFromAdvancePayment(dueDate);
        Iterable<List<LoanAccountData>> loansToBeRepaid = Iterables.partition(loansToBeRepaidData, batchSize);
        final Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
        List<Callable<Object>> advancePaymentPosters = new ArrayList<Callable<Object>>();
        
        for (List<LoanAccountData> subList : loansToBeRepaid) {
        	AdvancePaymentPoster poster = (AdvancePaymentPoster) this.applicationContext.getBean("advancePaymentPoster");
        	poster.setLoans(subList);
        	poster.setTenant(ThreadLocalContextUtil.getTenant());
        	poster.setAuthentication(authentication);
        	poster.setCommandService(commandService);
        	advancePaymentPosters.add(Executors.callable(poster));
        }

        try {
			List<Future<Object>> responses = executor.invokeAll(advancePaymentPosters);
for(Future f : responses) {
                    	f.get();
                    }
		} catch (InterruptedException e1) {
			e1.printStackTrace();
		}
        
        executor.shutdown(); 

 ```

  was:
For starters, it will be useful to have some technical configuration added as job parameter for each of the below jobs, that is batch size and thread pool size for executor service so this has a dependency on FINERACT-425. 

Add migration script to add those 2 configurations to each of the following jobs with the value of batch size being 500 and thread pool size being 16 :-

Add Accrual Transactions
Add Periodic Accrual Transactions
Add Accrual Transactions For Loans With Income Posted As Transactions
Generate Loan Loss Provisioning
Post Interest for Savings
Update Loan Summary

This would require separation of core functionality to a separate class.

Simplified example :-

final ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
        
        final LocalDate dueDate = DateUtils.getLocalDateOfTenant();
        final Collection<LoanAccountData> loansToBeRepaidData = this.loanReadPlatformService
                .retrieveLoansToBeRepaidFromAdvancePayment(dueDate);
        Iterable<List<LoanAccountData>> loansToBeRepaid = Iterables.partition(loansToBeRepaidData, batchSize);
        final Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
        List<Callable<Object>> advancePaymentPosters = new ArrayList<Callable<Object>>();
        
        for (List<LoanAccountData> subList : loansToBeRepaid) {
        	AdvancePaymentPoster poster = (AdvancePaymentPoster) this.applicationContext.getBean("advancePaymentPoster");
        	poster.setLoans(subList);
        	poster.setTenant(ThreadLocalContextUtil.getTenant());
        	poster.setAuthentication(authentication);
        	poster.setCommandService(commandService);
        	advancePaymentPosters.add(Executors.callable(poster));
        }

        try {
			List<Future<Object>> responses = executor.invokeAll(advancePaymentPosters);
for(Future f : responses) {
                    	f.get();
                    }
		} catch (InterruptedException e1) {
			e1.printStackTrace();
		}
        
        executor.shutdown(); 

 
> Parallelization of Jobs
> -----------------------
>
>                 Key: FINERACT-428
>                 URL: https://issues.apache.org/jira/browse/FINERACT-428
>             Project: Apache Fineract
>          Issue Type: Improvement
>          Components: Loan, Savings
>            Reporter: Avik Ganguly
>            Assignee: Markus Geiss
>              Labels: performance
>
> For starters, it will be useful to have some technical configuration added as job parameter for each of the below jobs, that is batch size and thread pool size for executor service so this has a dependency on FINERACT-425. 
> Add migration script to add those 2 configurations to each of the following jobs with the value of batch size being 500 and thread pool size being 16 :-
> Add Accrual Transactions
> Add Periodic Accrual Transactions
> Add Accrual Transactions For Loans With Income Posted As Transactions
> Generate Loan Loss Provisioning
> Post Interest for Savings
> Update Loan Summary
> This would require separation of core functionality to a separate class.
> Simplified example :-
> ```
> final ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
>         
>         final LocalDate dueDate = DateUtils.getLocalDateOfTenant();
>         final Collection<LoanAccountData> loansToBeRepaidData = this.loanReadPlatformService
>                 .retrieveLoansToBeRepaidFromAdvancePayment(dueDate);
>         Iterable<List<LoanAccountData>> loansToBeRepaid = Iterables.partition(loansToBeRepaidData, batchSize);
>         final Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
>         List<Callable<Object>> advancePaymentPosters = new ArrayList<Callable<Object>>();
>         
>         for (List<LoanAccountData> subList : loansToBeRepaid) {
>         	AdvancePaymentPoster poster = (AdvancePaymentPoster) this.applicationContext.getBean("advancePaymentPoster");
>         	poster.setLoans(subList);
>         	poster.setTenant(ThreadLocalContextUtil.getTenant());
>         	poster.setAuthentication(authentication);
>         	poster.setCommandService(commandService);
>         	advancePaymentPosters.add(Executors.callable(poster));
>         }
>         try {
> 			List<Future<Object>> responses = executor.invokeAll(advancePaymentPosters);
> for(Future f : responses) {
>                     	f.get();
>                     }
> 		} catch (InterruptedException e1) {
> 			e1.printStackTrace();
> 		}
>         
>         executor.shutdown(); 
>  ```


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)