db-torque-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YannickR <Yannick.Rich...@matricis.com>
Subject Re: LargeSelect example ?
Date Fri, 26 Oct 2007 13:14:04 GMT


YannickR wrote:
> 
> 
> YannickR wrote:
>> 
>> 
>> Yannick Richard wrote:
>>> 
>>> 
>>> Thomas Vandahl-2 wrote:
>>>> 
>>>> Yannick Richard wrote:
>>>>> Hi,
>>>>> 
>>>>>  
>>>>> 
>>>>> I am currently working on a Torque project that will handle database
>>>>> synchronization. 
>>>>> 
>>>>> The problem we have is an Out of Memory exception while selecting a
>>>>> big
>>>>> bunch of data from the database.
>>>>> 
>>>>>  
>>>>> 
>>>>> Here is the command we are using :
>>>>> 
>>>>> List ObjectsFromDB = ObjectPeer.doSelect(criteria, connection);
>>>>> 
>>>>>  
>>>>> 
>>>>> I saw the LargeSelect class you worked on but cannot find any Java
>>>>> example that could help me go forward.
>>>>> 
>>>>> Could you help me point to an example or help me understand how to
>>>>> integrate LargeSelect ?
>>>> 
>>>> Just a few hints, I don't have a complete example at hand:
>>>> 
>>>> 	LargeSelect ls = new LargeSelect(criteria, pageSize,
>>>> 				memoryPageLimit,
>>>> 				ObjectPeer.class.getName());
>>>> 
>>>> where the pageSize defines how many records to get with one call and
>>>> the
>>>> memoryPageLimit defines how many of these pages to "read ahead".
>>>> 
>>>> With this object you can now loop through the pages and LargeSelect
>>>> will
>>>> load the necessary data as needed, (pageSize * memoryPageLimit) records
>>>> at a time. Like:
>>>> 
>>>> 	while (ls.getNextResultsAvailable())
>>>> 	{
>>>> 		List ObjectsFromDB = ls.getNextResults();
>>>> 		// do what is necessary
>>>> 	}
>>>> 
>>>> See the JavaDoc at
>>>> http://db.apache.org/torque/releases/torque-3.3/runtime/apidocs/index.html
>>>> for more information.
>>>> 
>>>> Bye, Thomas.
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
>>>> For additional commands, e-mail: torque-user-help@db.apache.org
>>>> 
>>>> 
>>>> 
>>> 
>>> With your explanations I succeeded in running LargeSelect. It is now
>>> working while reading 10 000 records with a PageSize of 1000 and a
>>> MemoryPageLimit of 20. I don't know if it is normal, but when I use a
>>> MemoryPageLimit of 5 (5x1000 records), it is reading 1 - 5000 of 10000
>>> again and again and never get out of the loop...
>>> 
>>> Next, I tried to use PageSize of 5000/MemoryPageLimit of 50 in order to
>>> read less than 250 000 records, I had following exception : Exception in
>>> thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
>>> 
>>> I then modified the Eclipse shortcut arguments to better manage Heap
>>> memory, etc :
>>> -vmargs -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -Xms128M -Xmx1024M
>>> 
>>> Unfortunately, it jammed on first call of .getNextResults() for a
>>> night...
>>> When I debug it I can see that the Thread is sleeping in
>>> getResults(start, size) for all night on following command: while
>>> (((start + size - 1) > currentlyFilledTo) && !queryCompleted)
>>> 
>>> Is this a memory problem or I did something wrong ?
>>> 
>>> Regards,
>>> Yannick Richard
>>> 
>>> 
>>> 
>> 
> 
> Never mind, I found the solution :jumping:
> Readind Torque documentation, I tought MSSQL was not supported with Limit
> and Offset features used by LargeSelect so I installed TORQUE-84 patch
> from https://issues.apache.org/jira/browse/TORQUE-84 and this was causing
> getNextResultsAvailable() to not function correctly...
> 
> I was also misunderstanding the use of PageSize and MemoryPageLimit. Total
> batch size in memory is not only the number of records in a page but
> PageSize * MemoryPageLimit... 
> 
> So without that patch, that is not completed anyway, and some time spent
> on RTFM... Everything is fine now ! Thanks for your help !  
>  
> 
-- 
View this message in context: http://www.nabble.com/LargeSelect-example---tf4605414.html#a13427019
Sent from the Apache DB - Torque Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
For additional commands, e-mail: torque-user-help@db.apache.org


Mime
View raw message