From notifications-return-25832-archive-asf-public=cust-asf.ponee.io@ofbiz.apache.org Thu May 2 13:39:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 05D4E180671 for ; Thu, 2 May 2019 15:39:02 +0200 (CEST) Received: (qmail 35162 invoked by uid 500); 2 May 2019 13:39:02 -0000 Mailing-List: contact notifications-help@ofbiz.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ofbiz.apache.org Delivered-To: mailing list notifications@ofbiz.apache.org Received: (qmail 35150 invoked by uid 99); 2 May 2019 13:39:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 May 2019 13:39:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5E711E0104 for ; Thu, 2 May 2019 13:39:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 78D882581F for ; Thu, 2 May 2019 13:39:00 +0000 (UTC) Date: Thu, 2 May 2019 13:39:00 +0000 (UTC) From: "Taher Alkhateeb (JIRA)" To: notifications@ofbiz.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/OFBIZ-10592?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Taher Alkhateeb updated OFBIZ-10592: ------------------------------------ Environment: (was: Two instances Ofbiz installation on two machines= , connected to an Apache=C2=A0HTTPD instance which acts as a proxy. Request to ofbiz instances are handled and load balanced. OFBiz version : 13.07.03 with Multitenant enabled OS: Ubuntu Linux 16.04 LTS RDBMS: MariaDB v10.1.24) > OutOfMemory and stucked JobPoller issue > --------------------------------------- > > Key: OFBIZ-10592 > URL: https://issues.apache.org/jira/browse/OFBIZ-10592 > Project: OFBiz > Issue Type: Bug > Components: ALL APPLICATIONS > Affects Versions: Release Branch 13.07 > Reporter: Giulio Speri > Assignee: Giulio Speri > Priority: Critical > Attachments: Screenshot from 2019-04-20 02-32-37.png, alloc_tree_= 600k_12102018.png, jvm_ofbiz1_profi_telem.png, jvm_prof_ofbiz1_telem2.png, = ofbiz1_jvm_profil_nojobpoller.png, recorder_object_600k_12102018.png, telem= etry_ovrl_600k_12102018.png > > > =C2=A0 > This installation is composed by two instances of OFBiz (v13.07.03), serv= ed via an Apache Tomcat webserver, along with a load balancer. > The database server is MariaDB. > =C2=A0 > We had the first problems, about 3 weeks ago, when suddenly, the front1 (= ofbiz instance 1), stopped serving web requests; front2, instead, was still= working correctly. > =C2=A0 > Obviously we checked the log files, and we saw that async services were f= ailing; the failure was accompanied by this error line: > =C2=A0 > *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead li= mit exceeded_* > =C2=A0 > We analyzed the situation with our system specialists, and they told us t= hat the application was highly stressing machine resources (cpu always at o= r near 100%, RAM usage rapidly increasing), until the jvm run out of memory= . > This "resource-high-consumption situation", occurred only when ofbiz1 ins= tance was started with the JobPoller enabled; if the JobPoller was not enab= led, ofbiz run with low resource usage.=C2=A0 > =C2=A0 > We then focused on the db, to check first of all the dimensions; the resu= lt was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (abou= t 18 GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (ab= out 2 GB). > All the other tables had a size in the order of few MB each. > =C2=A0 > The first thing we did, was to clear all those tables, reducing considera= bly the db size. > After the cleaning, we tried to start ofbiz1 again, with the JobPoller co= mponent enabled; this caused a lot of old scheduled/queued jobs, to execute= . > Except than for the start-up time, the resource usage of the machine, sta= bilized around normal to low values (cpu 1-10%). > Ofbiz seemed to work (web request was served), but we noticed that the Jo= bPoller did not schedule or run jobs, anymore.=C2=A0 > The number of job in "Pending" state in the JobSandbox entity was small (= about 20); no Queued, no Failed, no jobs in other states. > In addition to this, unfortunately, after few hours, jvm run out of memor= y again. > =C2=A0 > Our jvm has an heap maximum size of 20GB ( we have 32GB on the=C2=A0 mach= ine), so it's not so small, I think. > The next step we're going to do is set-up locally the application over th= e same production db to see what happens. > =C2=A0 > Now that I explained the situation, I am going to ask if, in your opinion= /experience: > =C2=A0 > Could the JobPoller component be the root (and only) cause of the OutOfMe= mory of the jvm? > =C2=A0 > Could this issue be related to OFBIZ-5710? > =C2=A0 > Dumping and analyzing the heap of the jvm could help in some way to under= stand what or who fills the memory or is this operation a waste of time? > =C2=A0 > Is there something that we did not considered or missed during the whole = process of problem analysis? > =C2=A0 > =C2=A0 > I really thank you all for your attention and your help; any suggestion o= r advice would really be greatly appreciated. > =C2=A0 > Kind regards, > Giulio -- This message was sent by Atlassian JIRA (v7.6.3#76005)