lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fette, Georg" <georg.fe...@uni-wuerzburg.de>
Subject Re: Solr Server crashes when requesting a result with too large resultRows
Date Tue, 31 Jul 2018 09:12:42 GMT
Hi Andrea,
I agree that receiving too much data in one request is bad. But I was 
surprised that the query works with a lower but still very large rows 
parameter and that there is a threshold at which it crashes the server. 
Furthermore, it seems that the reason for the crash is not the size of 
the actual results because those are only 581.
Greetings
Georg

Am 31.07.2018 um 10:53 schrieb Andrea Gazzarini:
> Hi Georg,
> I would say, without knowing your context, that this is not what Solr 
> is supposed to do. You're asking to load everything in a single 
> request/response and this poses a problem.
> Since I guess that, even we assume it works, you should then iterate 
> those results one by one or in blocks, an option would be to do this 
> part (block scrolling) using Solr [2].
> I suggest you to have a look at
>
>  * the export endpoint [1]
>  * the cursor API [2]
>
> Best,
> Andrea
>
> [1] https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> [2] 
> https://lucene.apache.org/solr/guide/6_6/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors
>
> On 31/07/18 10:44, Georg Fette wrote:
>> Hello,
>> We run the server version 7.3.1. on a machine with 32GB RAM in a mode 
>> having -10g.
>> When requesting a query with
>> q={!boost 
>> b=sv_int_catalog_count_document}string_catalog_aliases:(*2*)&fq=string_field_type:catalog_entry&rows=2147483647
>> the server takes all available memory up to 10GB and is then no 
>> longer accessible with one processor at 100%.
>> When we reduce the rows parameter to 10000000 the query works. The 
>> query returns only 581 results.
>> The documentation at 
>> https://wiki.apache.org/solr/CommonQueryParameters states that as the 
>> "rows" parameter a "ridiculously large value" may be used, but this 
>> could pose a problem. The number we used was Int.max from Java.
>> Greetings
>> Georg
>>
>
>

-- 
---------------------------------------------------------------------
Dipl.-Inf. Georg Fette      Raum: B009
Universität Würzburg        Tel.: +49-(0)931-31-85516
Am Hubland                  Fax.: +49-(0)931-31-86732
97074 Würzburg              mail: georg.fette@uni-wuerzburg.de
---------------------------------------------------------------------


Mime
View raw message