lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to IW#commit
Date Fri, 02 Sep 2011 21:49:10 GMT

    [ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096346#comment-13096346
] 

Yonik Seeley commented on SOLR-2565:
------------------------------------

So now that commits no longer block adds, we should revisit what the best defaults are.
The transaction logging in SOLR-2700 needs to keep track of uncommitted documents - hence
if it doesn't affect performance too much, we should probably commit more often some how.
 Autocommit based on number of documents doesn't work well across the broad spectrum of users
(what value would work well for twitter indexers and book indexers).  A size-based approach
would probably work best, but we don't have that.  Maybe a time based approach?  That would
limit the transaction log size to the number of documents indexable in a given time period,
which should be roughly proportional to the document size.  I guess such a time period should
be somewhere between 10 and 60 seconds?  A lot of data can be indexed in 60 sec, and the goal
is to limit the transaction log size while not impacting performance too much due to increased
commit frequency.

The other issue is soft commits... should we configure a soft commitWithin by default (prob
within the range of 1-10 sec)?


> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
>                 Key: SOLR-2565
>                 URL: https://issues.apache.org/jira/browse/SOLR-2565
>             Project: Solr
>          Issue Type: Improvement
>          Components: update
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch,
SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193

> The main goal here is to prevent solr from closing the IW and use IW#commit instead.
AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what
we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can
have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup
issue for.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message