lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Hellman <jhell...@innoventsolutions.com>
Subject Re: openSearcher, default commit settings
Date Mon, 02 Jun 2014 18:46:00 GMT
Boon,

I expect you will find many definitions of “proper usage” depending upon context and expected
results.  Personally,  don’t believe this is Solr’s job to enforce, and there are many
ways through the use of directives in the servlet container layer that can allow restrictions
if you feel this is required.

I would recommend considering an abstraction layer if you feel your development team may (accidentally)
abuse the system they are permitted to use.  I’ve seen this employed very well with minimal
latency and cost in extremely large corporations that have many multiple development teams
using the same search infrastructure.

Jason

On Jun 2, 2014, at 3:53 AM, Boon Low <boon.low@dctfh.com> wrote:

> Thanks for clearing this up. The wiki, being an authoritative reference, needs to be
corrected.
> 
> Re. default commit settings. I agree educating developers is very essential. But in reality,
you can't rely on this as the sole mechanism for ensuring proper usage of the update API,
especially for calls such as "commit", "optimize", "expungeDeletes" which can be very expensive
for large indexes on a shared infrastructure.
> 
> The issue is, there's no control mechanism in Solr for update calls (cf. rewriting calls
via load-balancer). Once you expose the update handler to the developers, they could send
10 commit/optimise op per min, opening new searchers for each of those calls (openSearcher
is only configurable for autocommit). And there is nothing you can do about it in Solr, even
as a immediate stopgap while a fix is being implemented for the next sprint.
> 
> It's be good to have some consistency in terms of configuring handlers, i.e. having "default/invariant"
settings for both the search and update handlers.
> 
> Thanks,
> 
> Boon
> 
> 
> -----
> Boon Low
> Search Engineer, DCT Family History
> 
> On 29 May 2014, at 18:03, Shawn Heisey <solr@elyograg.org<mailto:solr@elyograg.org>>
wrote:
> 
> On 5/29/2014 9:21 AM, Boon Low wrote:
> 1. openSearcher (autoCommit)
> According to the Apache Solr reference, "autoCommit/openSearcher" is set to false by
default.
> 
> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
> 
> But on Solr v4.8.1, if "openSearcher" is omitted from the autoCommit config, new searchers
are opened and warmed post auto-commits. Is this behaviour intended or the wiki wrong?
> 
> I am reasonably certain that the default for openSearcher if it is not
> specified will always be true. My understanding and your actual
> experience says that the documentation is wrong.  Additional note: The
> docs for autoSoftCommit are basically a footnote on autoCommit, which I
> think is a mistake -- it should have its own section, and the docs
> should mention that openSearcher does not apply.
> 
> I think the code confirms this.  From SolrConfig.java:
> 
> protected UpdateHandlerInfo loadUpdatehandlerInfo() {
>   return new UpdateHandlerInfo(get("updateHandler/@class",null),
>           getInt("updateHandler/autoCommit/maxDocs",-1),
>           getInt("updateHandler/autoCommit/maxTime",-1),
>           getBool("updateHandler/autoCommit/openSearcher",true),
>           getInt("updateHandler/commitIntervalLowerBound",-1),
>           getInt("updateHandler/autoSoftCommit/maxDocs",-1),
>           getInt("updateHandler/autoSoftCommit/maxTime",-1),
>           getBool("updateHandler/commitWithin/softCommit",true));
> }
> 
> 2. openSearcher and other default commit settings
> From previous posts, I know it's not possible to disable commits completely in Solr config
(without coding). But is there a way to configure the default settings of hard/explicit commits
for the update handler? If not it makes sense to have a configuration mechanism. Currently,
a simple commit call seems to be hard-wired with the following options:
> 
> .. commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 
> There's no server-side option, e.g. to set "openSearcher=false" as default or "invariant"
(cf. searchHandler) to prevent new searchers from opening.
> 
> I found that at times it is necessary to have better server- or infrastructure-side controls
for update/commits, especially in agile teams. Client/UI developers do not necessarily have
complete Solr knowledge. Unintended commits from misbehaving client-side updates may be norm
(e.g. 10 times per minute!).
> 
> Since you want to handle commits automatically, you'll want to educate
> your developers and tell them that they should never send commits -- let
> Solr handle it.  If the code that talks to Solr is Java and uses SolrJ,
> you might want to consider using forbidden-apis in your project so that
> a build will fail if the commit method gets used.
> 
> https://code.google.com/p/forbidden-apis/
> 
> Thanks,
> Shawn
> 
> 
> 
> 
> 
> ______________________________________________________________________
> "brightsolid" is used in this email to mean brightsolid online technology
> limited.
> 
> Email Disclaimer
> 
> This message is confidential and may contain privileged information. You should
> not disclose its contents to any other person. If you are not the intended
> recipient, please notify the sender named above immediately. It is expressly
> declared that this e-mail does not constitute nor form part of a contract or
> unilateral obligation. Opinions, conclusions and other information in this
> message that do not relate to the official business of brightsolid shall be
> understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System. Powered by
> McAfee.
> ______________________________________________________________________


Mime
View raw message