lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-5038) Don't call MergePolicy / IndexWriter during DWPT Flush
Date Mon, 10 Jun 2013 19:18:21 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13679791#comment-13679791
] 

Shai Erera commented on LUCENE-5038:
------------------------------------

I get what you're saying but I reviewed the patch and I think it shows how more complex it
has become. It's not enough to set useCFS(true) on MP, but you also need to set it to IWC.
So now we have 3 settings: mp.setUseCFS(), mp.setNoCFSRatio() and iwc.setUseCFSOnWrite. That's
too much, no?

Worse thing is, if I do want to control CFS, how can I tell that IWC is not enough? At least
when everything is on MP, or on any other class I don't care, I get a hint from auto-complete
or reading jdocs.

I understand that the two settings are different, but the API is already there today and the
changes don't simplify it IMO. If it's a concurrency thing you want to solve, IW / DWPT can
read that setting in the ctor and never access MP again. I prefer to leave the API clear.
                
> Don't call MergePolicy / IndexWriter during DWPT Flush
> ------------------------------------------------------
>
>                 Key: LUCENE-5038
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5038
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 4.3
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>             Fix For: 4.4
>
>         Attachments: LUCENE-5038.patch, LUCENE-5038.patch, LUCENE-5038.patch
>
>
> We currently consult the indexwriter -> merge policy to decide if we need to write
CFS or not which is bad in many ways.
> - we should call mergepolicy only during merges
> - we should never sync on IW during DWPT flush
> - we should be able to make the decision if we need to write CFS or not before flush,
ie. we could write parts of the flush directly to CFS or even start writing stored fields
directly.
> - in the NRT case it might make sense to write all flushes to CFS to minimize filedescriptors
independent of the index size.
> I wonder if we can use a simple boolean for this in the IWC and get away with not consulting
merge policy. This would simplify concurrency a lot here already.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message