openjpa-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Woods <dwo...@apache.org>
Subject Re: Performance with large lists
Date Fri, 18 Jun 2010 19:54:44 GMT


On 6/18/10 12:47 PM, Adam Hardy wrote:
> 1) I'm using tomcat, so no Java EE container. I'm using Spring to handle
> the transactions.
> 
> 2) I'm using one EMF for the whole app from app start, and one EM for
> the transaction or sometimes 2, launched by a tomcat filter which opens
> it at the start of the HTTP request and closes it at the end. And the
> two transactions I'm talking about are 2 distinct user operations, so
> each gets one EM.
> 
> 3) Build-time enhancement.
> 
> 4) Actually no because in OpenJPA 1.2.2 there is a bug that throws NPEs
> and means I can't run with it enabled. I just upgraded to 2.0.0 I think
> I've got the same bug - albeit a different stack trace - but a quick
> google didn't show up any immediately available work-around. I guess
> this is the key issue, right?
> 

Which JIRA and what does your 2.0.0 stack trace look like?

> 
> 
> Here's my persistence xml
>   <persistence-unit name="OpenJpaJdbc">
>     <description>Pattern Repo JPA Config with OpenJPA</description>
>    
> <provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
> 
> <mapping-file>org/permacode/atomic/domain/entity/AtomicEntity.xml</mapping-file>
> 
>    
> <mapping-file>org/permacode/patternrepo/orm/BrokerAccount.xml</mapping-file>
> 
>     <mapping-file>org/permacode/patternrepo/orm/Category.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/DollarReturn.xml</mapping-file>
>     <mapping-file>org/permacode/patternrepo/orm/Exchange.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/ExchangeSymbol.xml</mapping-file>
> 
>     <mapping-file>org/permacode/patternrepo/orm/Fill.xml</mapping-file>
>     <mapping-file>org/permacode/patternrepo/orm/Market.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/MarketSymbol.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/MarketSystem.xml</mapping-file>
>     <mapping-file>org/permacode/patternrepo/orm/Order.xml</mapping-file>
>     <mapping-file>org/permacode/patternrepo/orm/Pattern.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/Portfolio.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/PortfolioItem.xml</mapping-file>
> 
>    
> <mapping-file>org/permacode/patternrepo/orm/RequiredParam.xml</mapping-file>
> 
>    
> <mapping-file>org/permacode/patternrepo/orm/TestAnalysis.xml</mapping-file>
>     <mapping-file>org/permacode/patternrepo/orm/TestRun.xml</mapping-file>
>     <mapping-file>org/permacode/patternrepo/orm/Trade.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/TradingParam.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/TradingSystem.xml</mapping-file>
> 
>    
> <mapping-file>org/permacode/patternrepo/orm/Weighting.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/WeightingGroup.xml</mapping-file>
> 
>    
> <mapping-file>org/permacode/patternrepo/orm/MiniAnalysis.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/MiniResult.xml</mapping-file>
>    
> <mapping-file>org/permacode/patternrepo/orm/TestQueries.xml</mapping-file>
>     <properties>
>       <property name="openjpa.RuntimeUnenhancedClasses"
> value="unsupported" />
>       <property name="openjpa.Log" value="SQL=TRACE"/>
>       <property name="openjpa.ConnectionFactoryProperties"
>         value="PrettyPrint=true, PrettyPrintLineLength=72" />
>     </properties>
>   </persistence-unit>
> 
> 
> 
> 
> Donald Woods on 18/06/10 13:36, wrote:
>> Some quick initial questions -
>> 1) Are you using Java SE or a Java EE container (more of a transactions
>> and XA question)?
>> 2) Are you using one EMF and EM for both the parent, child and
>> grandchild operations?  Are you calling flush() anywhere?
>> 3) Are you using build time or run time enhancement of entities?
>> 4) Have you enabled the DataCache or QueryCache?
>> http://openjpa.apache.org/builds/2.0.0/apache-openjpa-2.0.0/docs/manual/manual.html#ref_guide_cache_query
>>
>>
>> Also, what do the entities and your persistence.xml look like?
>>
>>
>> -Donald
>>
>>
>> On 6/18/10 8:11 AM, Adam Hardy wrote:
>>> I have a transaction that submits around 20K records, followed by
>>> another transaction in the user process which inserts the same into a
>>> related table.
>>>
>>> The performance issues I talked about below initially caused processing
>>> times of 45 mins, but I worked on the Java in the code and I tweaked the
>>> mySQL database and reduced it to 15 mins.
>>>
>>> This though is still a problem - I've been over the optimization
>>> guidelines in the documentation and there's nothing there that I can
>>> implement that I'm not already.
>>>
>>> The first transaction I mentioned takes 5 mins, but the second takes
>>> 15mins and is inserting child records of the records created in the
>>> first transaction. It looks like OpenJPA is fetching all of those first
>>> records again. Shouldn't they already be in memory?
>>>
>>>
>>> Thanks
>>> Adam
>>>
>>> Adam Hardy on 12/06/10 13:21, wrote:
>>>> I am trying to get a handle on what I should be able to achieve.
>>>>
>>>> Can someone give me some idea of the metrics I should be able to get
>>>> optimistically when persisting an object that has a child list with
>>>> 20,000 child objects and 20,000 grandchildren? (one-to-one child ->
>>>> grandchild)
>>>>
>>>> Can I reasonably expect to get this done in under a minute?
>>>>
>>>> I think that would work out at a rate of about 1.5 milliseconds per
>>>> object.
>>>>
>>>> Thanks
>>>> Adam
>>>>
>>>> Adam Hardy on 11/06/10 17:34, wrote:
>>>>> I have performance problems with large lists of beans due to the base
>>>>> class I am using for my entities.
>>>>>
>>>>> This is slightly non-OpenJPA specific, so I hope nobody minds, but it
>>>>> is Friday afternoon so I'm hoping you give me a bit of slack here.
>>>>>
>>>>> The problem arises when I start building lists with over 10,000 items
>>>>> on a parent class.
>>>>>
>>>>> The trouble is in the base class for the entities, which is quite
>>>>> clever (but obviously not clever enough) and it has non-facile
>>>>> equals() and hashcode() algorithms which make use of reflection. It's
>>>>> here that the slow-down comes.
>>>>>
>>>>> When I link the child with a parent that already has 10,000 children,
>>>>> the equals() method is called by ArrayList before the new child is
>>>>> placed in the index.
>>>>>
>>>>> As far as I can tell I have a couple of options.
>>>>>
>>>>> (1) ditch the reflection-based equals method and hard-code an equals
>>>>> method.
>>>>>
>>>>> (2) don't use ArrayList but find a Collection-based class that uses
>>>>> hashes or similar to identify items instead of equals. This is just
>>>>> speculation - perhaps there is no such thing or it wouldn't help
>>>>> anyway:
>>>>>
>>>>>     - would a collection using hashes caches the hashes of the items
>>>>> already indexed?
>>>>>    - would such a collection be persistable?
>>>>>
>>>>> If anyone has  been in this situation before, or has an idea about it
>>>>> , I'd really appreciate the help.
>>>
>>
> 
> 

Mime
View raw message