db-torque-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YannickR <Yannick.Rich...@matricis.com>
Subject Re: LargeSelect example ?
Date Thu, 25 Oct 2007 14:08:23 GMT


Yannick Richard wrote:
> 
> 
> Thomas Vandahl-2 wrote:
>> 
>> Yannick Richard wrote:
>>> Hi,
>>> 
>>>  
>>> 
>>> I am currently working on a Torque project that will handle database
>>> synchronization. 
>>> 
>>> The problem we have is an Out of Memory exception while selecting a big
>>> bunch of data from the database.
>>> 
>>>  
>>> 
>>> Here is the command we are using :
>>> 
>>> List ObjectsFromDB = ObjectPeer.doSelect(criteria, connection);
>>> 
>>>  
>>> 
>>> I saw the LargeSelect class you worked on but cannot find any Java
>>> example that could help me go forward.
>>> 
>>> Could you help me point to an example or help me understand how to
>>> integrate LargeSelect ?
>> 
>> Just a few hints, I don't have a complete example at hand:
>> 
>> 	LargeSelect ls = new LargeSelect(criteria, pageSize,
>> 				memoryPageLimit,
>> 				ObjectPeer.class.getName());
>> 
>> where the pageSize defines how many records to get with one call and the
>> memoryPageLimit defines how many of these pages to "read ahead".
>> 
>> With this object you can now loop through the pages and LargeSelect will
>> load the necessary data as needed, (pageSize * memoryPageLimit) records
>> at a time. Like:
>> 
>> 	while (ls.getNextResultsAvailable())
>> 	{
>> 		List ObjectsFromDB = ls.getNextResults();
>> 		// do what is necessary
>> 	}
>> 
>> See the JavaDoc at
>> http://db.apache.org/torque/releases/torque-3.3/runtime/apidocs/index.html
>> for more information.
>> 
>> Bye, Thomas.
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: torque-user-help@db.apache.org
>> 
>> 
>> 
> 
> With your explanations I succeeded in running LargeSelect. It is now
> working while reading 10 000 records with a PageSize of 1000 and a
> MemoryPageLimit of 20. I don't know if it is normal, but when I use a
> MemoryPageLimit of 5 (5x1000 records), it is reading 1 - 5000 of 10000
> again and again and never get out of the loop...
> 
> Next, I tried to use PageSize of 5000/MemoryPageLimit of 50 in order to
> read less than 250 000 records, I had following exception : Exception in
> thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
> 
> I then modified the Eclipse shortcut arguments to better manage Heap
> memory, etc :
> -vmargs -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -Xms128M -Xmx1024M
> 
> Unfortunately, it jammed on first call of .getNextResults() for a night...
> When I debug it I can see that the Thread is sleeping in getResults(start,
> size) for all night on following command: while (((start + size - 1) >
> currentlyFilledTo) && !queryCompleted)
> 
> Is this a memory problem or I did something wrong ?
> 
> Regards,
> Yannick Richard
> 
> 
> 
-- 
View this message in context: http://www.nabble.com/LargeSelect-example---tf4605414.html#a13407505
Sent from the Apache DB - Torque Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
For additional commands, e-mail: torque-user-help@db.apache.org


Mime
View raw message