lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (LUCENE-3653) Lucene Search not scalling
Date Sun, 18 Dec 2011 01:07:30 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171726#comment-13171726
] 

Uwe Schindler edited comment on LUCENE-3653 at 12/18/11 1:05 AM:
-----------------------------------------------------------------

Based on my work for another issue I added removal of synchronization for the caches in VirtualMethod
and AttributeSource. As those caches are generally read (should be without synchronization)
and seldom populated, ConcurrentHashMap is the best choice, as gets are without locks.
The problem is that those reflection caches need a WeakHashMap, but there is no ConcurrentWeakHashMap.
Based on the work on issue LUCENE-3531, I reactivated my WeakIdentityMap and made the backing
map exchangeable (there are 2 static factory methods allowing ConcurrentHashMap or simple
HashMaps as backing map). For the reflection caches, identity is also fine (Class.equals/hashcode
is based on class identity). The reflection caches also have no problem doing the heavy reflection
work twice if two threads are requesting the same class at the same time - work is done twice
and stored twice in map, but who cares?

This patch improves concurrency on creation of TokenStreams and instantiation of all classes
using VirtualMethod for backwards compatibility.
                
      was (Author: thetaphi):
    Based on my work for another issue I added removal of synchronization for the caches in
VirtualMethod and AttributeSource. As those caches are generally read (should be without synchronization)
and seldom populated, ConcurrentHashMap is the best choice, as gets are without locks.
The problem is that those reflection caches need a WeakHashMap, but there is no ConcurrentWeakHashMap.
Based on the work on issue LUCENE-3531, I reactivated my WeakIdentityMap and made the backing
map exchangeable (there are 2 static factory methods allowing ConcurrentHashMap or simple
HasMaps as backing map). For the reflection caches, identity is also fine (Class.equals/hashcode
is based on identity). The reflection caches also have no problem, doing the work twice if
two threads are requesting the same class at the same time - work is done twice and stored
twice in map, but who cares?

This patch improves concurrency on creation of TokenStreams and all classes using VirtualMethod
for backwards compatibility.
                  
> Lucene Search not scalling
> --------------------------
>
>                 Key: LUCENE-3653
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3653
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Gerrit Jansen van Vuuren
>         Attachments: App.java, LUCENE-3653-VirtualMethod+AttributeSource.patch, profile_1_a.png,
profile_1_b.png, profile_1_c.png, profile_1_d.png, profile_2_a.png, profile_2_b.png, profile_2_c.png
>
>
> I've noticed that when doing thousands of searches in a single thread the average time
is quite low i.e. a few milliseconds. When adding more concurrent searches doing exactly the
same search the average time increases drastically. 
> I've profiled the search classes and found that the whole of lucene blocks on 
> org.apache.lucene.index.SegmentCoreReaders.getTermsReader
> org.apache.lucene.util.VirtualMethod
>   public synchronized int getImplementationDistance 
> org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces
> These cause search times to increase from a few milliseconds to up to 2 seconds when
doing 500 concurrent searches on the same in memory index. Note: That the index is not being
updates at all, so not refresh methods are called at any stage.
> Some questions:
>   Why do we need synchronization here?
>   There must be a non-lockable solution for these, they basically cause lucene to be
ok for single thread applications but disastrous for any concurrent implementation.
> I'll do some experiments by removing the synchronization from the methods of these classes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message