lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3653) Lucene Search not scalling
Date Sun, 18 Dec 2011 08:48:30 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171800#comment-13171800
] 

Uwe Schindler commented on LUCENE-3653:
---------------------------------------

bq. I'm not clear on this one, you mean every RAMFile is opened once per search? or Will be
reused across all searches? If the later all threads will block on RAMFile always, this is
not minor but major, especially taking into account that I want to move from 200 concurrent
requests to 8000.

Once per opening IndexReader, during searches there is no file open/closing done at all. Synchronization
on the directory of files is only done when writing the files, and only when opening files
during IndexReader openm, but not on every access. When files are deleted there are other
contention points, but not during searches, as IndexReader is opened R/O.
                
> Lucene Search not scalling
> --------------------------
>
>                 Key: LUCENE-3653
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3653
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Gerrit Jansen van Vuuren
>         Attachments: App.java, LUCENE-3653-VirtualMethod+AttributeSource.patch, LUCENE-3653-VirtualMethod+AttributeSource.patch,
lucene-unsync.diff, profile_1_a.png, profile_1_b.png, profile_1_c.png, profile_1_d.png, profile_2_a.png,
profile_2_b.png, profile_2_c.png
>
>
> I've noticed that when doing thousands of searches in a single thread the average time
is quite low i.e. a few milliseconds. When adding more concurrent searches doing exactly the
same search the average time increases drastically. 
> I've profiled the search classes and found that the whole of lucene blocks on 
> org.apache.lucene.index.SegmentCoreReaders.getTermsReader
> org.apache.lucene.util.VirtualMethod
>   public synchronized int getImplementationDistance 
> org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces
> These cause search times to increase from a few milliseconds to up to 2 seconds when
doing 500 concurrent searches on the same in memory index. Note: That the index is not being
updates at all, so not refresh methods are called at any stage.
> Some questions:
>   Why do we need synchronization here?
>   There must be a non-lockable solution for these, they basically cause lucene to be
ok for single thread applications but disastrous for any concurrent implementation.
> I'll do some experiments by removing the synchronization from the methods of these classes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message