lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lord Khan Han <khanuniver...@gmail.com>
Subject Shingle and Query Performance
Date Fri, 26 Aug 2011 21:49:23 GMT
Hi,

We are indexing news  document from the various sites. Currently we have
200K docs indexed. Total index size is 36 gig.  There is also attachement to
the news (pdf -docs etc) So document size could be high (ie 10mb).

We are using some complex queries which includes around 30 - 40 terms per
query. %70 of this terms is two word phrases. We are using
with conjunction +  and -  to pinpoint exact result.
There is also grouping, dismax and boosting , Termvector HL  .

Our problem is query times. Currently its around 6-7 secs. I know our query
is little bit heavy but we want to improve query performance. I believe we
can make it sub second but no succes at the moment.

We tried to use shingle 2 word token it decreases the query performcen !! We
assumed it will help the speed up phrases search..  What could be
your suggestions ? What we are missing.

(using solr latest trunk and HW is pretty good, 32 core  with 32 gig ram)

Here the field def:

<fieldType name="sh_text" class="solr.TextField" positionIncrementGap="100"
autoGeneratePhraseQueries="true">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
        <!--<filter class="solr.LowerCaseFilterFactory"/>-->
        <filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
        <filter class="solr.ShingleFilterFactory" maxShingleSize="2"
outputUnigrams="true"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <!--<filter class="solr.LowerCaseFilterFactory"/>-->
        <filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
        <filter class="solr.ShingleFilterFactory" maxShingleSize="2"
outputUnigrams="true"/>
      </analyzer>
    </fieldType>

and

 <field name="content" type="sh_text" stored="true" indexed="true"
termVectors="true" termPositions="true" termOffsets="true"/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message