cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aristedes Maniatis <...@maniatis.org>
Subject Re: Transactions , Primary Keys
Date Sun, 23 Sep 2007 08:19:55 GMT

On 23/09/2007, at 2:23 PM, Philip Copeland wrote:

> Did you have an issue committing every "several thousand records".  
> What
> would you do if it failed at that stage - did you figure out a way to
> resume again at a known point? Handling large imports is never easy.

Initially we had some memory issues, but improvements by Andrus in  
the builds of Cayenne 3 around April 2007 fixed that nicely. Then we  
experimented with different batch sizes and their effect on speed. We  
found that 1000 worked well for us, so I think that's what we left it  
at in the end.

We prevent failure during commit by running validateForSave() on each  
record. If it fails we remove it from the context and write out a log  
entry. In our case we regularly see hundreds of errors in the import  
data, but thankfully they mostly aren't our problem to fix. That way  
we continue right to the end, even if some records fail. This allows  
the customer to fix the errors and either import them along, or  
return to a backup database and rerun the whole thing.

Of course, your goals may be quite different.

Ari Maniatis


-------------------------->
Aristedes Maniatis
phone +61 2 9660 9700
PGP fingerprint 08 57 20 4B 80 69 59 E2  A9 BF 2D 48 C2 20 0C C8



Mime
View raw message