db-torque-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Vandahl ...@apache.org>
Subject Re: CriteriaEnumeration
Date Sat, 10 Mar 2007 10:45:04 GMT
Thoralf Rickert wrote:
> Hi!
> I'm not sure, if this was discussed before, but I want to send you this
> "extension". 
> Currently I'm working on somekind of "backup-procedure" that reads some
> data from a database and creates a complex XML file (for example: stores
> with addresses and products). To do this, I have to get all "stores"
> from the database. If there are a lot of stores and I make just a
> doSelect(new Criteria()) I have (of course) trouble with the memory
> (number of stores > 100000).
> I could implement a special algorithm, but I decided to implement a
> mechanism that takes a Criteria and set offset/limit internally in a
> loop. In this way I can get 100 stores with one SELECT and do my work.
> If the "pool" is empty, I can read the next 100 stores.

This is more or less what LargeSelect does. I used to use it for such
things - reading data one page at a time. It provides the additional
advantage of a "read-ahead-cache", means it continues reading your data
in the background while you can process the first page of records.

If you are interested, I created a small LargeSelectHelper class which
allows you to continuously read one page at a time and let LargeSelect
do the "chunking".

Bye, Thomas.

To unsubscribe, e-mail: torque-user-unsubscribe@db.apache.org
For additional commands, e-mail: torque-user-help@db.apache.org

View raw message