lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Rafalovitch <arafa...@gmail.com>
Subject Re: Fail to huge collection extraction
Date Sun, 09 Sep 2012 04:05:07 GMT
I think the point here is the question about your use of the data.

If you want to show it to the client, then you are unlikely to need
details of more than 1 screenful of records (e.g. 10). When user goes
to another screen, you rerun the query and specify values 11-20, etc.
SOLR does not have a problem rerunning complex queries and returning
different subset of results.

On the other hand, if you are not presenting this to the user directly
and do need all records at once, perhaps you should not be pulling all
records details from SOLR, but just use it for search. That is, let
SOLR return just the primary keys of the matches and you can then send
a request to a dedicated database with the list of IDs. Databases and
drives are specifically designed around giving streaming results
without crashing/timing-out. SOLR is a search system and is not
perfect as a retrieval system or primary system of record (though it
is getting there slowly).

Hope this helps.

Regards,
   Alex.

Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
book)


On Sat, Sep 8, 2012 at 11:20 PM, neosky <neosky11@yahoo.com> wrote:
> I am sorry that I can't get your point. Would you explain a little more?
> I am still struggling with this problem. It seems crash by no meaning
> sometimes. Even I reduce to 5000 records each time, but sometimes it works
> well with 10000 per page.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Fail-to-huge-collection-extraction-tp4003559p4006399.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message