lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to IW#commit
Date Fri, 19 Aug 2011 01:44:29 GMT

    [ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087453#comment-13087453
] 

Hoss Man commented on SOLR-2565:
--------------------------------

Summary of IRC discussion (as i remember it) that didn't make it to the mailing list...
* Use and rmuir found that they could consistently reproduce "slow" and "fast" behavior in
some tests using fixed seeds...
** {{ant test -Dtestcase=SolrExampleBinaryTest -Dtests.seed=1:1:1}} ... always slow
** {{ant test -Dtestcase=SolrExampleBinaryTest -Dtests.seed=2:2:2}} ... always fast
* for Uwe and sarowe (on windows), slow ment "taking so long they finally just had to give
up and kill the JVM"
* For rmuir on his linux box, slow vs. fast was ~1-10 seconds vs 100 seconds
* For hossman on my linux box, results were roughly consistent with rmuir, but slightly slower
since i have a crappier box
* with the path Uwe commited in r1159448 rmuir, uwe, and sarowe all agreed the tests were
"fast" again
* testUnicode (where the majority of the slowness is seen) is one of the few test methods
in these slow classes that uses random testing in a loop, so the compounded speed issues there
are likely from some cumulative slow down that is typically only done rarely in other tests
(likely "commit" given the nature of the commit that seemed to cause the problem)
* Jenkin's is using a test multiplier of "3" which is specifically used in testUnicode to
pick the number of iterations, so that obviously compounds the effects of this slowdown in
jenkins -- but makes me highly suspicious about why Uwe and sarowe were seeing so much longer
test times on their windows machines then rmuir or i were seeing on linux (since they weren't
using any special test multiplier)



> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
>                 Key: SOLR-2565
>                 URL: https://issues.apache.org/jira/browse/SOLR-2565
>             Project: Solr
>          Issue Type: Improvement
>          Components: update
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch,
dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193

> The main goal here is to prevent solr from closing the IW and use IW#commit instead.
AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what
we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can
have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup
issue for.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message