lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerrit Jansen van Vuuren (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3653) Lucene Search not scalling
Date Sat, 17 Dec 2011 19:39:30 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171649#comment-13171649
] 

Gerrit Jansen van Vuuren commented on LUCENE-3653:
--------------------------------------------------

Hi Uwe,

Thanks for the fast response. Maybe it would help you to understand my use case:

-------------- Start ------------------------
I receive a customer request.
The create a Query based on the customer data.
Search a Lucene index in memory, not disk activity is done. i.e. I use RAMDirectory.
Then return the response to the customer.

With a single request I get on average 3ms response times.
If I ramp it up to 200 requests I start getting 200-500ms response times.
------------ End ------------------------
For my application every millisecond counts, just does and out of my control.

Assumptions:
 (1)  I create a new Instance of Query on each Request. The Query object is not shared ever.
So, no locking is needed and no thread blocking should ever happen.
 (2)  The index is read only, and in memory so no OS file locking is applicable or will ever
occur.
 (3) No index refresh happens and no writing, this is purely read only.

Point is no need for locking anywhere.


   




 



                
> Lucene Search not scalling
> --------------------------
>
>                 Key: LUCENE-3653
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3653
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Gerrit Jansen van Vuuren
>         Attachments: App.java, profile_1_a.png, profile_1_b.png, profile_1_c.png, profile_1_d.png,
profile_2_a.png, profile_2_b.png, profile_2_c.png
>
>
> I've noticed that when doing thousands of searches in a single thread the average time
is quite low i.e. a few milliseconds. When adding more concurrent searches doing exactly the
same search the average time increases drastically. 
> I've profiled the search classes and found that the whole of lucene blocks on 
> org.apache.lucene.index.SegmentCoreReaders.getTermsReader
> org.apache.lucene.util.VirtualMethod
>   public synchronized int getImplementationDistance 
> org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces
> These cause search times to increase from a few milliseconds to up to 2 seconds when
doing 500 concurrent searches on the same in memory index. Note: That the index is not being
updates at all, so not refresh methods are called at any stage.
> Some questions:
>   Why do we need synchronization here?
>   There must be a non-lockable solution for these, they basically cause lucene to be
ok for single thread applications but disastrous for any concurrent implementation.
> I'll do some experiments by removing the synchronization from the methods of these classes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message