openjpa-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Hardy <adam....@cyberspaceroad.com>
Subject Re: Performance with large lists
Date Fri, 18 Jun 2010 16:47:09 GMT
1) I'm using tomcat, so no Java EE container. I'm using Spring to handle the 
transactions.

2) I'm using one EMF for the whole app from app start, and one EM for the 
transaction or sometimes 2, launched by a tomcat filter which opens it at the 
start of the HTTP request and closes it at the end. And the two transactions I'm 
talking about are 2 distinct user operations, so each gets one EM.

3) Build-time enhancement.

4) Actually no because in OpenJPA 1.2.2 there is a bug that throws NPEs and 
means I can't run with it enabled. I just upgraded to 2.0.0 I think I've got the 
same bug - albeit a different stack trace - but a quick google didn't show up 
any immediately available work-around. I guess this is the key issue, right?



Here's my persistence xml
   <persistence-unit name="OpenJpaJdbc">
     <description>Pattern Repo JPA Config with OpenJPA</description>
     <provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
 
<mapping-file>org/permacode/atomic/domain/entity/AtomicEntity.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/BrokerAccount.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Category.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/DollarReturn.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Exchange.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/ExchangeSymbol.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Fill.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Market.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/MarketSymbol.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/MarketSystem.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Order.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Pattern.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Portfolio.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/PortfolioItem.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/RequiredParam.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/TestAnalysis.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/TestRun.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Trade.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/TradingParam.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/TradingSystem.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/Weighting.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/WeightingGroup.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/MiniAnalysis.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/MiniResult.xml</mapping-file>
     <mapping-file>org/permacode/patternrepo/orm/TestQueries.xml</mapping-file>
     <properties>
       <property name="openjpa.RuntimeUnenhancedClasses" value="unsupported" />
       <property name="openjpa.Log" value="SQL=TRACE"/>
       <property name="openjpa.ConnectionFactoryProperties"
         value="PrettyPrint=true, PrettyPrintLineLength=72" />
     </properties>
   </persistence-unit>




Donald Woods on 18/06/10 13:36, wrote:
> Some quick initial questions -
> 1) Are you using Java SE or a Java EE container (more of a transactions
> and XA question)?
> 2) Are you using one EMF and EM for both the parent, child and
> grandchild operations?  Are you calling flush() anywhere?
> 3) Are you using build time or run time enhancement of entities?
> 4) Have you enabled the DataCache or QueryCache?
> http://openjpa.apache.org/builds/2.0.0/apache-openjpa-2.0.0/docs/manual/manual.html#ref_guide_cache_query
> 
> Also, what do the entities and your persistence.xml look like?
> 
> 
> -Donald
> 
> 
> On 6/18/10 8:11 AM, Adam Hardy wrote:
>> I have a transaction that submits around 20K records, followed by
>> another transaction in the user process which inserts the same into a
>> related table.
>>
>> The performance issues I talked about below initially caused processing
>> times of 45 mins, but I worked on the Java in the code and I tweaked the
>> mySQL database and reduced it to 15 mins.
>>
>> This though is still a problem - I've been over the optimization
>> guidelines in the documentation and there's nothing there that I can
>> implement that I'm not already.
>>
>> The first transaction I mentioned takes 5 mins, but the second takes
>> 15mins and is inserting child records of the records created in the
>> first transaction. It looks like OpenJPA is fetching all of those first
>> records again. Shouldn't they already be in memory?
>>
>>
>> Thanks
>> Adam
>>
>> Adam Hardy on 12/06/10 13:21, wrote:
>>> I am trying to get a handle on what I should be able to achieve.
>>>
>>> Can someone give me some idea of the metrics I should be able to get
>>> optimistically when persisting an object that has a child list with
>>> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
>>> grandchild)
>>>
>>> Can I reasonably expect to get this done in under a minute?
>>>
>>> I think that would work out at a rate of about 1.5 milliseconds per
>>> object.
>>>
>>> Thanks
>>> Adam
>>>
>>> Adam Hardy on 11/06/10 17:34, wrote:
>>>> I have performance problems with large lists of beans due to the base
>>>> class I am using for my entities.
>>>>
>>>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>>>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>>>
>>>> The problem arises when I start building lists with over 10,000 items
>>>> on a parent class.
>>>>
>>>> The trouble is in the base class for the entities, which is quite
>>>> clever (but obviously not clever enough) and it has non-facile
>>>> equals() and hashcode() algorithms which make use of reflection. It's
>>>> here that the slow-down comes.
>>>>
>>>> When I link the child with a parent that already has 10,000 children,
>>>> the equals() method is called by ArrayList before the new child is
>>>> placed in the index.
>>>>
>>>> As far as I can tell I have a couple of options.
>>>>
>>>> (1) ditch the reflection-based equals method and hard-code an equals
>>>> method.
>>>>
>>>> (2) don't use ArrayList but find a Collection-based class that uses
>>>> hashes or similar to identify items instead of equals. This is just
>>>> speculation - perhaps there is no such thing or it wouldn't help anyway:
>>>>
>>>>     - would a collection using hashes caches the hashes of the items
>>>> already indexed?
>>>>    - would such a collection be persistable?
>>>>
>>>> If anyone has  been in this situation before, or has an idea about it
>>>> , I'd really appreciate the help.
>>
> 


Mime
View raw message