Mailing-List: contact dev-help@fineract.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@fineract.incubator.apache.org
MIME-Version: 1.0
Date: Thu, 26 May 2016 13:22:00 +0530
Message-ID: <CAOSe2ysjwLAfDjJUzHy=KhoGv95Caph4M-CmjQHbEar4e8HqSA@mail.gmail.com>
Subject: [Fineract] Scheduler Jobs - Lock Acquisition Issue
From: Nazeer Shaik <nazeer.shaik@confluxtechnologies.com>
To: dev@fineract.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1145b3caaead3c0533ba1003
archived-at: Thu, 26 May 2016 07:52:07 -0000

--001a1145b3caaead3c0533ba1003
Content-Type: text/plain; charset=UTF-8

Hi Devs,

*We are facing one issue with our scheduler jobs and your help is
appreciated. *

We have Scheduler Jobs which runs at configured time for each tenant. Each
job falls into a job group. At server boot up, we are creating Scheduler
instances for each job group. While triggering any job, corresponding
scheduler instance will be picked up based on job's group id.

Before triggering any job, we are updating it's running status to true in
DB, so that no other thread runs same job parellelly.
Before updating the running status, we are locking it by using
LockModeType.PESSIMISTIC_WRITE.

The problem what we are facing is, for some reason we are not able to
acquire the lock and we are getting Lock Acquisition exception. After this,
all triggers (existing/new) in that group scheduler entering into BLOCKED
state. The only way is to recover is restart the server. Please find issue
description here https://issues.apache.org/jira/browse/FINERACT-145

Now we are catching that exception outside of transaction scope. After
doing this, non of the trigger(s) are entering into blocked state. Since we
have to run this job we have added code to retry acquiring the lock for
some time. If we are not able to acquire the lock, we will veto the
execution.

Please find modified code below in
org.apache.fineract.infrastructure.jobs.service.*SchedulerTriggerListener*.

@Override
public boolean vetoJobExecution(final Trigger trigger, final
JobExecutionContext context) {
        final String tenantIdentifier =
trigger.getJobDataMap().getString(SchedulerServiceConstants.TENANT_IDENTIFIER);
        final FineractPlatformTenant tenant =
this.tenantDetailsService.loadTenantById(tenantIdentifier);
        ThreadLocalContextUtil.setTenant(tenant);
        final JobKey key = trigger.getJobKey();
        final String jobKey = key.getName() +
SchedulerServiceConstants.JOB_KEY_SEPERATOR + key.getGroup();
        String triggerType = SchedulerServiceConstants.TRIGGER_TYPE_CRON;
        if
(context.getMergedJobDataMap().containsKey(SchedulerServiceConstants.TRIGGER_TYPE_REFERENCE))
{
            triggerType =
context.getMergedJobDataMap().getString(SchedulerServiceConstants.TRIGGER_TYPE_REFERENCE);
        }

        Integer maxNumberOfRetries = ThreadLocalContextUtil.getTenant()
                .getConnection().getMaxRetriesOnDeadlock();
        Integer maxIntervalBetweenRetries =
ThreadLocalContextUtil.getTenant()
                .getConnection().getMaxIntervalBetweenRetries();
        Integer numberOfRetries = 0;
        boolean proceedJobExecution = false ;
        while (numberOfRetries <= maxNumberOfRetries) {
            try {
                proceedJobExecution =
this.schedularService.processJobDetailForExecution(jobKey, triggerType);
//In above method call we are checking whether it is currently running or
not. If not running we will lock it and update the running //status
                numberOfRetries = maxNumberOfRetries + 1;
            } catch (LockAcquisitionException exception) {
                logger.debug("Not able to acquire the lock to update job
running status for JobKey: "+jobKey);
                try {
                    Random random = new Random();
                    int randomNum =
random.nextInt(maxIntervalBetweenRetries + 1);
                    Thread.sleep(1000 + (randomNum * 1000));
                    numberOfRetries = numberOfRetries + 1;
                } catch (InterruptedException e) {

                }
            }
        }
        return proceedJobExecution ;
    }

Can you please review this solution and let me know if you have any other
approach to fix this.

Thanks,
Nazeer

--001a1145b3caaead3c0533ba1003--