lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shalin Shekhar Mangar (JIRA)" <>
Subject [jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
Date Fri, 16 Sep 2016 03:38:20 GMT


Shalin Shekhar Mangar commented on SOLR-9521:

Thanks Russel this is very useful!

I wonder if we can get rid of specifying the log file names in solrconfig.xml or at least
make them optional. Now that Solr has got rid of the war and we control Jetty and logging,
it should be possible to automatically detect the logging directory and use the recent log

bq. We have a socketTimeout of 1500 ms on the SolrClient, a socketTimeout of 1000 ms on the

Please be sure to use the "async" parameter for all collection APIs because the same shardHandlerFactory
is used for all collection admin requests as well as the internal core admin requests performed
as part of the collection APIs.

To Shawn's point of optimize -- those use the update shard handler and won't be affected by

> Warm first searcher with queries from log file
> ----------------------------------------------
>                 Key: SOLR-9521
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: search
>    Affects Versions: 6.2
>            Reporter: Russell Black
>            Priority: Minor
>              Labels: performance
>         Attachments:
> The only way I know of to warm the first searcher before it is opened, is to hard code
warming queries in
> {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our more expensive
filter queries change periodically.  
> I have created a plugin that warms the first searcher from the most recent entries in
the log files.  In my opinion this capability ought to be part of solr, so I'm contributing
my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its "queries" argument
from recent queries from the log files.  It is configured like this:
> {code:xml}
> <listener event="firstSearcher" class="solrplugin.LogFileWarmer">
>   <!-- The arguments shown below are the defaults and could be omitted -->
>   <!-- A list of filenames to search.  They are searched in the order
>        listed, and the lines in each file are searched in reverse
>        order, so that the most recent queries are used to warm the
>        cache.  Searching stops when it has gathered enough queries.
>   -->
>   <str name="file">logs/solr.log</str>
>   <str name="file">logs/solr.log.1</str>
>   <str name="file">logs/solr.log.2</str>
>   <str name="file">logs/solr.log.3</str>
>   <str name="file">logs/solr.log.4</str>
>   <str name="file">logs/solr.log.5</str>
>   <str name="file">logs/solr.log.6</str>
>   <str name="file">logs/solr.log.7</str>
>   <str name="file">logs/solr.log.8</str>
>   <str name="file">logs/solr.log.9</str>
>   <!-- The maximum number of queries to use from logs -->
>   <int name="count">100</int>
>   <!-- The maximum number of log lines to search in order to find <count> queries-->
>   <int name="maxLines">1000000</int>
>   <!-- The regular expression that matches the query string from the log line.  It
expects the query string to be in capture group 1. -->
>   <str name="regex">path=/select params=\{([^ ]+)\} </str>
>   <!-- if no queries can be found in the log files, fall back to this -->
>   <arr name="fallback">
>     <lst><str name="q">solr</str><str name="sort">price asc</str></lst>
>     <lst><str name="q">rocks</str><str name="sort">weight asc</str></lst>
>   </arr>
> <listener>
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message