cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrus Adamchik <and...@objectstyle.org>
Subject Re: OutOfMemory with large number of inserts
Date Mon, 24 May 2010 07:19:53 GMT
Actually you can do batch processing with Cayenne, as DataContext uses  
weak references to committed objects. So doing commit after each X  
inserted objects should release the memory. However "rel" object  
reference may prevent that from happening if it is the same object for  
all inserted objects. For one thing make sure there's no reverse  
relationship from rel to l's.

Andrus


On May 24, 2010, at 10:09 AM, Andrey Razumovsky wrote:
> ORM technologies require additional memory than just plain JDBC.  
> Million
> records you read will lead to creating million data objects and this  
> is not
> a case for ORM, because huge size of memory is needed. Moreover,  
> you'll see
> major slowdowns during the commit. So the actual advice is to use  
> plain SQL
> (or say Cayenne's SQLTemplate) for such great inserts
>
> 2010/5/24 Steve Wells <websystemz@gmail.com>
>
>> Hi All,
>>
>> I am importing from a lot of csv files (2200+) and each csv has  
>> between 0
>> and 3000 rows, each row will create a new DataObject in a  
>> DataContext etc :
>>
>> The code is simple enough and distilled here is:
>>
>> for each csv file
>>  RelatedObj rel = getRelatedObj()
>>  for each line in csv file
>>     Line l = new Line()
>>     l.setxxx for each column of csv     // 5 cols are set
>>     l.setToRelatedObject(rel)
>>     dataContext.registerNewObject(l)
>>  end for each line in csv file
>>  dataContext.commitChanges()
>> end
>>
>> This goes well for a while but no matter how high I set the heap  
>> size I
>> eventually get OutOfMemoryException....I can see the speed of each  
>> commit
>> slowdown over time as the heap size is diminshed.
>>
>> I have tried a few different ways around this but all end up the  
>> same.
>> Initially tuning the SQL server instance, but everything points to  
>> objects
>> in memory not being de-allocated.
>>
>> Has anyone had experience with Cayenne (3.0RC3) and this number of  
>> inserts
>> they could advise with?
>>
>> Cheers,
>>
>> Steve
>>
>
>
>
> -- 
> Andrey


Mime
View raw message