db-torque-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Monroe <Greg.Mon...@dukece.com>
Subject RE: Extensive memory usage in torque-generator. Bug?
Date Mon, 30 Nov 2009 19:25:13 GMT
Sorry for the delay... I was gone a week for US Thanksgiving
holiday.

You might look at Torque's sibling project called ddlutils,
http://db.apache.org/ddlutils This uses the same basic XML format
as Torque but specifically targeted at doing the data setup tasks 
that have been added into the torque core.  FWIW, one of the V4.0 
goals is to start using this project as the code behind these
tasks.

Also, I while back I wrote a couple of Add-ons to Torque that 
allowed a method to import and export torque objects as XML. This
consists of a generator addin that creates Betwixt maps for 
Torque data objects and a Runtime framework for using this to 
export and import data.  It's was released as a SourceForge 
project:  http://torque-addons.sourceforge.net/

If I remember right, one possible gotcha is that the Maven 2 
plug-in (due to Maven 2 issues) can't control the build path like
Ant and Maven 1.  So the betwixt add-in may have to be merged with
the Torque template jar and this substituted for the dependency.
Not sure if this is right since I use ant for my builds and am
not up on Maven.


> -----Original Message-----
> From: Ludwig Magnusson [mailto:ludwig@itcatapult.com]
> Sent: Tuesday, November 24, 2009 3:21 PM
> To: 'Apache Torque Developers List'
> Subject: RE: Extensive memory usage in torque-generator. Bug?
> 
> In reply to this, I actually found the maven option to extend the amount
> of
> memory.
> Now the VM can use 512Mb. I tried with my _actual_ data-xml file (which
> was
> created by torque:datadump) and that task succeeded but used 415Mb. My
> computer has 4Gb of RAM but extending the allocated memory does not seem
> like a solution. A database can grow much larger than the one we have.
> We will take a look at the generator code and see if there is anything we
> can find. (that would be the first place to look at right?)
> /Ludwig
> 
> 
> 
> -----Original Message-----
> From: Thomas Fischer [mailto:fischer@seitenbau.net]
> Sent: den 24 november 2009 21:09
> To: Apache Torque Developers List
> Subject: RE: Extensive memory usage in torque-generator. Bug?
> 
> Hm, on the one hand 65 MB RAM is not much on a modern computer. On the
> other hand, it is much as compared to the data, as you said.
> I am afraid that probably we do not have the manpower to dig into this;
> memory usage is difficult to track down. If you find a reason for the
> extensive memory usage and propose a solution, We'd be glad to
> incorporate
> it.
> 
> For your immediate problem, try increasing the maven memory limits. (the
> usual -Xmx -Xms options for the java vm).
> 
>      Thomas
> 
> 
> > A few days ago I posted in the users list that I got a
> > java.lang.OutOfMemoryError while trying to generate my data-sql.
> >
> > I did some stress testing on the generator, here is how it went:
> >
> >
> >
> > I used a simple table and tried to insert different number of rows,
> ranging
> > from 1 to 2000.
> >
> > This is my data.xml for n number of roles:
> >
> > <dataset name="all">
> >
> >   <Role Id="0" Name="Role_0"/>
> >
> >   <Role Id="1" Name="Role_1"/>
> >
> >   .
> >
> >   <Role Id="n" Name="Role_n"/>
> >
> > </dataset>
> >
> >
> >
> > At the end of each run maven prints the amount of memory used (final
> > memory), here are some values:
> >
> >
> >
> > 1 row - 11M/20M
> >
> > 100 rows - 15M/25M
> >
> > 1000 rows - 42M/63M
> >
> > 1500 rows - 58M/63M
> >
> > 2000 rows - fail 63M/63M
> >
> >
> >
> > The limit seems to be around 1500 rows.
> >
> > When that's the case, the input file is ca 60Kb and the output file is
> ca
> > 100Kb, so how can the generator use 58Mb of memory?
> >
> > Is this a bug? Known or unknown?
> >
> > /Ludwig
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: torque-dev-help@db.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: torque-dev-help@db.apache.org

DukeCE Privacy Statement:
Please be advised that this e-mail and any files transmitted with
it are confidential communication or may otherwise be privileged or
confidential and are intended solely for the individual or entity
to whom they are addressed. If you are not the intended recipient
you may not rely on the contents of this email or any attachments,
and we ask that you please not read, copy or retransmit this
communication, but reply to the sender and destroy the email, its
contents, and all copies thereof immediately. Any unauthorized
dissemination, distribution or copying of this communication is
strictly prohibited.

---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org


Mime
View raw message