lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (Commented) (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3653) Lucene Search not scalling
Date Sat, 17 Dec 2011 21:11:41 GMT


Uwe Schindler commented on LUCENE-3653:


short comment about your code and how to remove parts of the sync:
- The SegmentCoreReaders sync cannot be removed in Lucene 3.x, as segments are read/write.
You can remove the synchronization partly in *your* Lucene instance by patching it, the risk
is on your side! This is not a bug in Lucene. In Lucene trunk, most of this sync is removed
as IndexReaders will be pure read-only. We are currently working on removing contention in
SegmentReader's SegmentCore.
- You are not resusing the Analyzer. Create *one* analyzer instance and reuse it for indexing
and for QueryParser! When you do this, you will see no contention on TokenStream creation
(VirtualMethod.getImplementationDistance, AttributeSource.getAttributeInterfaces). TokenStream
creation by Analyzers is a heavy operation (not only the reflection cache contention), so
use only one Analyzer per app and pass it to all you QueryParsers. Removing sync from AttributeSource/VirtualMethod
caches will corrupt the cache and cause bugs - but thats unneeded if you reuse as already
> Lucene Search not scalling
> --------------------------
>                 Key: LUCENE-3653
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Gerrit Jansen van Vuuren
>         Attachments:, profile_1_a.png, profile_1_b.png, profile_1_c.png, profile_1_d.png,
profile_2_a.png, profile_2_b.png, profile_2_c.png
> I've noticed that when doing thousands of searches in a single thread the average time
is quite low i.e. a few milliseconds. When adding more concurrent searches doing exactly the
same search the average time increases drastically. 
> I've profiled the search classes and found that the whole of lucene blocks on 
> org.apache.lucene.index.SegmentCoreReaders.getTermsReader
> org.apache.lucene.util.VirtualMethod
>   public synchronized int getImplementationDistance 
> org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces
> These cause search times to increase from a few milliseconds to up to 2 seconds when
doing 500 concurrent searches on the same in memory index. Note: That the index is not being
updates at all, so not refresh methods are called at any stage.
> Some questions:
>   Why do we need synchronization here?
>   There must be a non-lockable solution for these, they basically cause lucene to be
ok for single thread applications but disastrous for any concurrent implementation.
> I'll do some experiments by removing the synchronization from the methods of these classes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message