Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of peter.sturge@gmail.com
 designates 209.85.223.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAN4YXveS8dWqEN7kpiLqUDPwrnKMNjdFmh2rs1kmW-yz7bCoMw@mail.gmail.com>
References: 
 <CADjjot_3p58souc9UD0wUdtBy4VFPwGvhUwZ5+=76uvbffdcJg@mail.gmail.com>
	<CADjjot-2K=Y0sCLZh1gBZ++fQt4mL4QTFepQ55vpiqhBHo-XJg@mail.gmail.com>
	<CAN4YXveS8dWqEN7kpiLqUDPwrnKMNjdFmh2rs1kmW-yz7bCoMw@mail.gmail.com>
Date: Sat, 29 Jun 2013 23:58:42 +0100
Message-ID: 
 <CALG2m7O82jAH3CEZabkV1f8FiLdxJF7Ac3D457cOsaxkL4d=7Q@mail.gmail.com>
Subject: Re: Improving performance to return 2000+ documents
From: Peter Sturge <peter.sturge@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=089e0122aa1ada497f04e052ef01

--089e0122aa1ada497f04e052ef01
Content-Type: text/plain; charset=ISO-8859-1

Hello Utkarsh,
This may or may not be relevant for your use-case, but the way we deal with
this scenario is to retrieve the top N documents 5,10,20or100 at a time
(user selectable). We can then page the results, changing the start
parameter to return the next set. This allows us to 'retrieve' millions of
documents - we just do it at the user's leisure, rather than make them wait
for the whole lot in one go.
This works well because users very rarely want to see ALL 2000 (or whatever
number) documents at one time - it's simply too much to take in at one time.
If your use-case involves an automated or offline procedure (e.g. running a
report or some data-mining op), then presumably it doesn't matter so much
it takes a bit longer (as long as it returns in some reasonble time).
Have you looked at doing paging on the client-side - this will hugely
speed-up your search time.
HTH
Peter


On Sat, Jun 29, 2013 at 6:17 PM, Erick Erickson <erickerickson@gmail.com>wrote:

> Well, depending on how many docs get served
> from the cache the time will vary. But this is
> just ugly, if you can avoid this use-case it would
> be a Good Thing.
>
> Problem here is that each and every shard must
> assemble the list of 2,000 documents (just ID and
> sort criteria, usually score).
>
> Then the node serving the original request merges
> the sub-lists to pick the top 2,000. Then the node
> sends another request to each shard to get
> the full document. Then the node merges this
> into the full list to return to the user.
>
> Solr really isn't built for this use-case, is it actually
> a compelling situation?
>
> And having your document cache set at 1M is kinda
> high if you have very big documents.
>
> FWIW,
> Erick
>
>
> On Fri, Jun 28, 2013 at 8:44 PM, Utkarsh Sengar <utkarsh2012@gmail.com
> >wrote:
>
> > Also, I don't see a consistent response time from solr, I ran ab again
> and
> > I get this:
> >
> > ubuntu@ip-10-149-6-68:~$ ab -c 10 -n 500 "
> >
> >
> http://x.amazonaws.com:8983/solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json
> > "
> >
> >
> > Benchmarking x.amazonaws.com (be patient)
> > Completed 100 requests
> > Completed 200 requests
> > Completed 300 requests
> > Completed 400 requests
> > Completed 500 requests
> > Finished 500 requests
> >
> >
> > Server Software:
> > Server Hostname:       x.amazonaws.com
> > Server Port:            8983
> >
> > Document Path:
> >
> >
> /solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json
> > Document Length:        1538537 bytes
> >
> > Concurrency Level:      10
> > Time taken for tests:   10.858 seconds
> > Complete requests:      500
> > Failed requests:        8
> >    (Connect: 0, Receive: 0, Length: 8, Exceptions: 0)
> > Write errors:           0
> > Total transferred:      769297992 bytes
> > HTML transferred:       769268492 bytes
> > Requests per second:    46.05 [#/sec] (mean)
> > Time per request:       217.167 [ms] (mean)
> > Time per request:       21.717 [ms] (mean, across all concurrent
> requests)
> > Transfer rate:          69187.90 [Kbytes/sec] received
> >
> > Connection Times (ms)
> >               min  mean[+/-sd] median   max
> > Connect:        0    0   0.3      0       2
> > Processing:   110  215  72.0    190     497
> > Waiting:       91  180  70.5    152     473
> > Total:        112  216  72.0    191     497
> >
> > Percentage of the requests served within a certain time (ms)
> >   50%    191
> >   66%    225
> >   75%    252
> >   80%    272
> >   90%    319
> >   95%    364
> >   98%    420
> >   99%    453
> >  100%    497 (longest request)
> >
> >
> > Sometimes it takes a lot of time, sometimes its pretty quick.
> >
> > Thanks,
> > -Utkarsh
> >
> >
> > On Fri, Jun 28, 2013 at 5:39 PM, Utkarsh Sengar <utkarsh2012@gmail.com
> > >wrote:
> >
> > > Hello,
> > >
> > > I have a usecase where I need to retrive top 2000 documents matching a
> > > query.
> > > What are the parameters (in query, solrconfig, schema) I shoud look at
> to
> > > improve this?
> > >
> > > I have 45M documents in 3node solrcloud 4.3.1 with 3 shards, with 30GB
> > > RAM, 8vCPU and 7GB JVM heap size.
> > >
> > > I have documentCache:
> > >   <documentCache class="solr.LRUCache"  size="1000000"
> > > initialSize="1000000"   autowarmCount="0"/>
> > >
> > > allText is a copyField.
> > >
> > > This is the result I get:
> > > ubuntu@ip-10-149-6-68:~$ ab -c 10 -n 500 "
> > >
> >
> http://x.amazonaws.com:8983/solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json
> > > "
> > >
> > > Benchmarking x.amazonaws.com (be patient)
> > > Completed 100 requests
> > > Completed 200 requests
> > > Completed 300 requests
> > > Completed 400 requests
> > > Completed 500 requests
> > > Finished 500 requests
> > >
> > >
> > > Server Software:
> > > Server Hostname:        x.amazonaws.com
> > > Server Port:            8983
> > >
> > > Document Path:
> > >
> >
> /solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json
> > > Document Length:        1538537 bytes
> > >
> > > Concurrency Level:      10
> > > Time taken for tests:   35.999 seconds
> > > Complete requests:      500
> > > Failed requests:        21
> > >    (Connect: 0, Receive: 0, Length: 21, Exceptions: 0)
> > > Write errors:           0
> > > Non-2xx responses:      2
> > > Total transferred:      766221660 bytes
> > > HTML transferred:       766191806 bytes
> > > Requests per second:    13.89 [#/sec] (mean)
> > > Time per request:       719.981 [ms] (mean)
> > > Time per request:       71.998 [ms] (mean, across all concurrent
> > requests)
> > > Transfer rate:          20785.65 [Kbytes/sec] received
> > >
> > > Connection Times (ms)
> > >               min  mean[+/-sd] median   max
> > > Connect:        0    0   0.6      0       8
> > > Processing:     9  717 2339.6    199   12611
> > > Waiting:        9  635 2233.6    164   12580
> > > Total:          9  718 2339.6    199   12611
> > >
> > > Percentage of the requests served within a certain time (ms)
> > >   50%    199
> > >   66%    236
> > >   75%    263
> > >   80%    281
> > >   90%    548
> > >   95%    838
> > >   98%  12475
> > >   99%  12545
> > >  100%  12611 (longest request)
> > >
> > > --
> > > Thanks,
> > > -Utkarsh
> > >
> >
> >
> >
> > --
> > Thanks,
> > -Utkarsh
> >
>

--089e0122aa1ada497f04e052ef01--