Mailing-List: contact torque-user-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Torque Users List" <torque-user@db.apache.org>
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <13430608.post@talk.nabble.com>
Date: Fri, 26 Oct 2007 09:26:14 -0700 (PDT)
From: YannickR <Yannick.Richard@matricis.com>
To: torque-user@db.apache.org
Subject: Re: LargeSelect example ?
In-Reply-To: <13427019.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: 
 <511ACAD1FA56E64E9866C5B5686A63AF38D2F1@SV-WINDJAMMER.matricis.local>
 <4710DB21.5060106@apache.org> <13407505.post@talk.nabble.com>
 <13427019.post@talk.nabble.com>


YannickR wrote:
> 
> 
> YannickR wrote:
>> 
>> 
>> YannickR wrote:
>>> 
>>> 
>>> YannickR wrote:
>>>> 
>>>> 
>>>> Yannick Richard wrote:
>>>>> 
>>>>> 
>>>>> Thomas Vandahl-2 wrote:
>>>>>> 
>>>>>> Yannick Richard wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> I am currently working on a Torque project that will handle database
>>>>>>> synchronization. 
>>>>>>> 
>>>>>>> The problem we have is an Out of Memory exception while selecting a
>>>>>>> big
>>>>>>> bunch of data from the database.
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> Here is the command we are using :
>>>>>>> 
>>>>>>> List ObjectsFromDB = ObjectPeer.doSelect(criteria, connection);
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> I saw the LargeSelect class you worked on but cannot find any Java
>>>>>>> example that could help me go forward.
>>>>>>> 
>>>>>>> Could you help me point to an example or help me understand how to
>>>>>>> integrate LargeSelect ?
>>>>>> 
>>>>>> Just a few hints, I don't have a complete example at hand:
>>>>>> 
>>>>>> 	LargeSelect ls = new LargeSelect(criteria, pageSize,
>>>>>> 				memoryPageLimit,
>>>>>> 				ObjectPeer.class.getName());
>>>>>> 
>>>>>> where the pageSize defines how many records to get with one call and
>>>>>> the
>>>>>> memoryPageLimit defines how many of these pages to "read ahead".
>>>>>> 
>>>>>> With this object you can now loop through the pages and LargeSelect
>>>>>> will
>>>>>> load the necessary data as needed, (pageSize * memoryPageLimit)
>>>>>> records
>>>>>> at a time. Like:
>>>>>> 
>>>>>> 	while (ls.getNextResultsAvailable())
>>>>>> 	{
>>>>>> 		List ObjectsFromDB = ls.getNextResults();
>>>>>> 		// do what is necessary
>>>>>> 	}
>>>>>> 
>>>>>> See the JavaDoc at
>>>>>> http://db.apache.org/torque/releases/torque-3.3/runtime/apidocs/index.html
>>>>>> for more information.
>>>>>> 
>>>>>> Bye, Thomas.
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
>>>>>> For additional commands, e-mail: torque-user-help@db.apache.org
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> With your explanations I succeeded in running LargeSelect. It is now
>>>>> working while reading 10 000 records with a PageSize of 1000 and a
>>>>> MemoryPageLimit of 20. I don't know if it is normal, but when I use a
>>>>> MemoryPageLimit of 5 (5x1000 records), it is reading 1 - 5000 of 10000
>>>>> again and again and never get out of the loop...
>>>>> 
>>>>> Next, I tried to use PageSize of 5000/MemoryPageLimit of 50 in order
>>>>> to read less than 250 000 records, I had following exception :
>>>>> Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap
>>>>> space
>>>>> 
>>>>> I then modified the Eclipse shortcut arguments to better manage Heap
>>>>> memory, etc :
>>>>> -vmargs -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -Xms128M -Xmx1024M
>>>>> 
>>>>> Unfortunately, it jammed on first call of .getNextResults() for a
>>>>> night...
>>>>> When I debug it I can see that the Thread is sleeping in
>>>>> getResults(start, size) for all night on following command: while
>>>>> (((start + size - 1) > currentlyFilledTo) && !queryCompleted)
>>>>> 
>>>>> Is this a memory problem or I did something wrong ?
>>>>> 
>>>>> Regards,
>>>>> Yannick Richard
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> Never mind, I found the solution :jumping:
>>> Readind Torque documentation, I tought MSSQL was not supported with
>>> Limit and Offset features used by LargeSelect so I installed TORQUE-84
>>> patch from https://issues.apache.org/jira/browse/TORQUE-84 and this was
>>> causing getNextResultsAvailable() to not function correctly...
>>> 
>>> I was also misunderstanding the use of PageSize and MemoryPageLimit.
>>> Total batch size in memory is not only the number of records in a page
>>> but PageSize * MemoryPageLimit... 
>>> 
>>> So without that patch, that is not completed anyway, and some time spent
>>> on RTFM... Everything is fine now ! Thanks for your help !  
>>>  
>>> 
>> 
> 
> Sorry for confusion...
> I tough I succeeded to run LargeSelect with SQL server 2005, but it
> fails... continuing to read even after there are no more records...
> I did install TORQUE-84 patch in order to support Limit/Offset.
> 
> My last post on the forum, talking about removing the Patch, is wrong...
> It only works when your specify PageSize * MemoryPageLimit that will cover
> all your records... If not, it will end reading before the end of the
> records.
> 
> Is the patch working or not ? The status on
> https://issues.apache.org/jira/browse/TORQUE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> seems to be unresolved...
> 
> Could someone help me on that one ?
> Can I still use LargeSelect with MSSQL 2005 ?
> 
> Regards,
> Yannick Richard
> 
-- 
View this message in context: http://www.nabble.com/LargeSelect-example---tf4605414.html#a13430608
Sent from the Apache DB - Torque Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
For additional commands, e-mail: torque-user-help@db.apache.org