lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Faceting Word Count
Date Mon, 06 Nov 2017 15:42:05 GMT
_Why_ do you want to get the word counts? Faceting on all of the
tokens for 100M docs isn't something Solr is ordinarily used for. As
Emir says it'll take a huge amount of memory. You can use one of the
function queries (termfreq IIRC) that will give you the count of any
individual term you have and will be very fast.

But getting all of the word counts in the index is probably not
something I'd use Solr for.

This may be an XY problem, you're asking how to do something specific
(X) without explaining what the problem you're trying to solve is (Y).
Perhaps there's another way to accomplish (Y) if we knew more about
what it is.

Best,
Erick



On Mon, Nov 6, 2017 at 4:15 AM, Emir Arnautović
<emir.arnautovic@sematext.com> wrote:
> Hi Wael,
> You are faceting on analyzed field. This results in field being uninverted - fieldValueCache
being built - on first call after every commit. This is both time and memory consuming (you
can check in admin console in stats how much memory it took).
> What you need to do is to create multivalue string field (not text) and parse values
(do analysis steps) on client side and store it like that. This will allow you to enable docValues
on that field and avoid building fieldValueCache.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 6 Nov 2017, at 13:06, Wael Kader <wael@softech-lb.com> wrote:
>>
>> Hi,
>>
>> I am using a custom field. Below is the field definition.
>> I am using this because I don't want stemming.
>>
>>
>>    <fieldType name="text_no_stem2" class="solr.TextField"
>> positionIncrementGap="100">
>>      <analyzer type="index">
>>        <charFilter class="solr.MappingCharFilterFactory"
>> mapping="mapping-ISOLatin1Accent.txt"/>
>>        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>
>>        <filter class="solr.StopFilterFactory"
>>                ignoreCase="true"
>>                words="stopwords.txt"
>>                enablePositionIncrements="true"
>>                />
>>        <filter class="solr.WordDelimiterFilterFactory"
>>                protected="protwords.txt"
>>                generateWordParts="0"
>>                generateNumberParts="1"
>>                catenateWords="1"
>>                catenateNumbers="1"
>>                catenateAll="0"
>>                splitOnCaseChange="1"
>>                preserveOriginal="1"/>
>>        <filter class="solr.LowerCaseFilterFactory"/>
>>
>>        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>      </analyzer>
>>      <analyzer type="query">
>>        <charFilter class="solr.MappingCharFilterFactory"
>> mapping="mapping-ISOLatin1Accent.txt"/>
>>        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>> ignoreCase="true" expand="true"/>
>>        <filter class="solr.StopFilterFactory"
>>                ignoreCase="true"
>>                words="stopwords.txt"
>>                enablePositionIncrements="true"
>>                />
>> <!--ORIGINAL                generateNumberParts="1"-->
>>        <filter class="solr.WordDelimiterFilterFactory"
>>                protected="protwords.txt"
>>                generateWordParts="0"
>>                catenateWords="0"
>>                catenateNumbers="0"
>>                catenateAll="0"
>>                splitOnCaseChange="1"
>>                preserveOriginal="1"/>
>>        <filter class="solr.LowerCaseFilterFactory"/>
>>        <!-- ORIGINAL filter class="solr.SnowballPorterFilterFactory"
>> language="English" protected="protwords.txt"/-->
>>        <!-- Webel: switch off Porter-stemmer algorithm to enforce whole
>> word match -->
>>        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>      </analyzer>
>>    </fieldType>
>>
>>
>> Regards,
>> Wael
>>
>> On Mon, Nov 6, 2017 at 10:29 AM, Emir Arnautović <
>> emir.arnautovic@sematext.com> wrote:
>>
>>> Hi Wael,
>>> Can you provide your field definition and sample query.
>>>
>>> Thanks,
>>> Emir
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>>
>>>
>>>
>>>> On 6 Nov 2017, at 08:30, Wael Kader <wael@softech-lb.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I am having an index with around 100 Million documents.
>>>> I have a multivalued column that I am saving big chunks of text data in.
>>> It
>>>> has around 20 GB of RAM and 4 CPU's.
>>>>
>>>> I was doing faceting on it to get word cloud but it was taking around 1
>>>> second to retrieve when the data was 5-10 Million .
>>>> Now I have more data and its taking minutes to get the results (that is
>>> if
>>>> it gets it and SOLR doesn't crash). Whats the best way to make it run or
>>>> maybe its not scalable to make it run on my current schema and design
>>> with
>>>> News articles.
>>>>
>>>> I am looking to find the best solution for this. Maybe create another
>>> index
>>>> to split the data while inserting it or maybe if I change some settings
>>> in
>>>> SolrConfig or add some RAM, it would perform better.
>>>>
>>>> --
>>>> Regards,
>>>> Wael
>>>
>>>
>>
>>
>> --
>> Regards,
>> Wael
>

Mime
View raw message