camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dr Immo Benjes <immoben...@locussoftware.com>
Subject Memory leak with enrich("sql:select * from WhatEver")
Date Wed, 30 Mar 2016 16:02:20 GMT

Hi 

I’ve been working with Camel for the last few month.

I have a program that processes several CVS files in order and inserts them into a DB. Once
they are all inserted I have to check for duplicate and create new CVS files without duplicates.

That is all working fine but I have one big problem - a memory leak.

I’ve tracked it down to a enrich(“sql”) component.

I have removed everything after that enrich and just added a to(“mock:result”).
Once the CVS files have been processed and inserted into the DB I send an exchange to 

from("seda:check_duplicates?concurrentConsumers=4")
.log("Checking for duplicates on ${header.NOTIFY_BY} for event ${header.EVENT_ID} #{header.SFTP_DIRECTORY}")
.enrich("sql:select NOTIFICATION_ID, NOTIFY_BY, NOTIFY_BY_VALUE, LANGUAGE, :#EVENT_ID as EVENT_ID,PDP_SR_PARTICIPANTS_PARTICIPANT_ID,EVENT_DISPLAY_PREMISE_ADDR,
CSV_LINE, count(*) as c from PDP_SR_NOTIFICATIONS where NOTIFY_BY = :#NOTIFY_BY and PDP_SR_EVENT_EVENT_ID
= :#EVENT_ID  group by PDP_SR_PARTICIPANTS_PARTICIPANT_ID, NOTIFY_BY_VALUE, LANGUAGE;") 
.to("mock:result");

Below is a screenshot of the memory consumption for two sets of CVS files. You can see the
jump in memory when the enrich(“sql”).

I would like to iterate over the result set (not load everything in one go as it can be quite
large but the useIterator=true does not seem to be supported (at least I’ve tried) and the
memory is never released again.

What am I doing wrong here? It is mainly chars in a HashMap that seems to be taking up the
memory which makes sense if the data is that of the sql query.


Regards

Immo
  
Dr. Immo Benjes
Senior Software Architect 



Castle Court
41 London Road
Reigate
Surrey RH2 9RJ
United Kingdom
Tel: +44 (0)1737 735012
Web: www.locussoftware.com
Email: immobenjes@locussoftware.com





Mime
View raw message