Mailing-List: contact user-help@cayenne.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cayenne.apache.org
Received-SPF: pass (herse.apache.org: domain of jean-paul.lefevre@cea.fr
 designates 132.166.172.103 as permitted sender)
From: Jean-Paul Le =?iso-8859-1?q?F=E8vre?= <jean-paul.lefevre@cea.fr>
Organization: CEA DAPNIA
To: user@cayenne.apache.org
Subject: Out of memory exception when creating a large number of objects
Date: Mon, 23 Jul 2007 15:29:35 +0200
User-Agent: KMail/1.9.4
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Message-Id: <200707231529.36162.jean-paul.lefevre@cea.fr>

Hi,

I'm trying to import a pretty big amount of data into my database.
The input is a xml formatted file. It describes more than 10 millions
of objects each having tens of attributes. The application parses the input=
=20
file, creates the cayenne objects and commits the changes if requested.

As you can imagining I'm facing difficulties trying to avoid out of memory
errors. Unfortunately, at this point, I'm still unable to load my big
input file.

To figure out what it's happening I'm monitoring the application behavior
with jconsole. My tactic is the following : every 10000 objects (this number
is a parameter) I call rollbackChanges() or commitChanges().

When I run the program in rollback mode It turns out that the memory used
oscillates between a min and a max value as expected : after each rollback
the garbage collector feels free to cleanup the memory.

But in commit mode the amount of memory keeps on increasing and the=20
application fails eventually.
I've tried to call unregisterNewObjects() and startTrackingNewObjects() aft=
er
the commit :

ctxt.commitChanges();
ctxt.getObjectStore().unregisterNewObjects();
ctxt.getEntityResolver().clearCache();
ctxt.getObjectStore().startTrackingNewObjects();

but it didn't help. It seems that cayenne keeps reference of newly created
objects somewhere preventing the gc from working.

Would you have an idea how to fix the problem ?
Thanks,


=2D-=20
___________________________________________________________________

Jean-Paul Le F=E8vre  * Mail : LeFevre@fonteny.org