lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to IW#commit
Date Sun, 04 Sep 2011 00:03:16 GMT

    [ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096774#comment-13096774
] 

Yonik Seeley commented on SOLR-2565:
------------------------------------

I did some performance tests w/ autoCommit=60000 (60 sec) vs no commits at all (10M small
docs)
6 runs for each configuration, times in seconds.

no commits (not even at the end): 202,222,217,206,203,196 average=207
autoCommit=60000: 201,206,210,226,220,201 average=210

So committing every 60 seconds actually came out faster (probably due to chance because of
the high variability).

Aside: there could be a race issue w/ autocommit.  I was checking the segments_n files to
verify that commits were actually taking place periodically.  On a new index, the first file
is segments_1.  All of the tests ran around 3 minutes 30 seconds... which should have meant
4 more commits, or a segments_5 file after 4 minutes.  I only checked a few times, but one
time I saw a segments_6, and two other times I saw a segments_9 file.  This suggests that
too many commits are happening for some reason.

> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
>                 Key: SOLR-2565
>                 URL: https://issues.apache.org/jira/browse/SOLR-2565
>             Project: Solr
>          Issue Type: Improvement
>          Components: update
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch,
SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193

> The main goal here is to prevent solr from closing the IW and use IW#commit instead.
AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what
we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can
have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup
issue for.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message