lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven White <swhite4...@gmail.com>
Subject Re: When is too many fields in "qf" is too many?
Date Wed, 20 May 2015 15:24:24 GMT
Thanks Shawn.

I have already switched to using POST because I need to send a long list of
data in "qf".  My question isn't about POST / GET, it's about Solr and
Lucene having to deal with such long list of fields.  Here is the text of
my question reposted:

> Given the above, beside the fact that a search for "apple" translating to
> a 20K characters passing over the network, what else within Solr and
Lucene
> I should be worried about if any?  Will I hit some kind of a limit?  Will
> each search now require more CPU cycles?  Memory?  Etc.

Steve



On Wed, May 20, 2015 at 10:52 AM, Shawn Heisey <apache@elyograg.org> wrote:

> On 5/20/2015 6:27 AM, Steven White wrote:
> > My solution requires that users in group-A can only search against a set
> of
> > fields-A and users in group-B can only search against a set of fields-B,
> > etc.  There can be several groups, as many as 100 even more.  To meet
> this
> > need, I build my search by passing in the list of fields via "qf".  What
> > goes into "qf" can be large: as many as 1500 fields and each field name
> > averages 15 characters long, in effect the data passed via "qf" will be
> > over 20K characters.
> >
> > Given the above, beside the fact that a search for "apple" translating
> to a
> > 20K characters passing over the network, what else within Solr and
> Lucene I
> > should be worried about if any?  Will I hit some kind of a limit?  Will
> > each search now require more CPU cycles?  Memory?  Etc.
>
> You have two choices when queries become that large.
>
> One is to increase the max HTTP header size in the servlet container.
> In most containers, webservers, and proxy servers, this defaults to 8192
> bytes.  This is an approach that works very well, but will not scale to
> extremely large sizes.  I have done this on my indexes, because I
> regularly have queries in the 20K range, but I do not expect them to get
> very much larger than this.
>
> The other option is to switch to sending a POST instead of a GET.  The
> default max POST size that Solr sets is 2MB, which is plenty for just
> about any query, and can be increased easily to much larger sizes.  If
> you are using SolrJ, switching to POST is very easy ... you'd need to
> research to figure out how if you're using another framework.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message