Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of erickerickson@gmail.com
 designates 209.85.219.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEFAe-E5NoOQwhAOQ-ncKJfs-nrfexg3eJB1swHu19=36edDSQ@mail.gmail.com>
References: <1346093786899-4003559.post@n3.nabble.com>
	<CAN4YXveL8wfCZYY6i9u6J4QZ7f6n5iPgB2MpXO+RTjkvxiUYSQ@mail.gmail.com>
	<1347160856135-4006399.post@n3.nabble.com>
	<CAEFAe-Gnr8b5yZRTwuEdAKuCmSB5w_W8pdkVJPjcSf0+ep2eqQ@mail.gmail.com>
	<1347204269308-4006450.post@n3.nabble.com>
	<CAEFAe-E5NoOQwhAOQ-ncKJfs-nrfexg3eJB1swHu19=36edDSQ@mail.gmail.com>
Date: Sun, 9 Sep 2012 12:10:17 -0700
Message-ID: 
 <CAN4YXvc4O+j3OEh8i0SQ92M7UD6bg3-O4KQCAWfJt6WsNFpxzA@mail.gmail.com>
Subject: Re: Fail to huge collection extraction
From: Erick Erickson <erickerickson@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Alexandre:

I'll buy you a beer sometime, it's just sooo pleasant when someone
else has the same worldview I do....

http://searchhub.org/dev/2011/11/03/stop-being-so-agreeable/

neosky:
Particularly look at the paragraph that has "the XY problem" in it.

Best
Erick

On Sun, Sep 9, 2012 at 8:56 AM, Alexandre Rafalovitch
<arafalov@gmail.com> wrote:
> I am sorry, but your customer is extremely unlikely to want the whole
> result in his browser. It is just a red flag that they are converting
> their (business) requirements into your (IT) language and that's what
> they end up with.
>
> Go the other way, ask them to pretend that you've done it already and
> then explain what happens once all those records are on their screen
> (and their operating system is no longer responsive :-) ). What is the
> business process that request is for. And how often they want to do
> this (and what is the significance of that frequency).
>
> Do they want a weekly audit copy to make sure nobody changed the
> records? Then, maybe they want a batch report emailed to them instead
> (or even just generated weekly on a shared drive). Do they want
> something they can access on their laptop while they are not connected
> to a network? Maybe they need a local replica of the (subset of the)
> app working from local index?
>
> Perhaps you have already asked that and this is just what they want.
> Then, I am afraid, you are just stuck fighting against the system
> designed for other use cases. Good luck.
>
> But if you haven't asked yet, do try! Do it often enough and you may
> get a payrise out of it because you will be meeting your clients on
> their territory instead of them having to come to yours.
>
> Regards,
>    Alex.
> Personal blog: http://blog.outerthoughts.com/
> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> - Time is the quality of nature that keeps events from happening all
> at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> book)
>
>
> On Sun, Sep 9, 2012 at 11:24 AM, neosky <neosky11@yahoo.com> wrote:
>> Thanks Alex!
>> Yes, you hit my key points.
>> Actually I have to implement both of the requirements.
>> The first one works very well as the reason you state. Now I have a website
>> client which is 20 records per page. It is fast.
>> However, my customer also wants to use Servlet to  download the whole query
>> set.(1 millions records maximum possible)
>> So at this time, I tried to use Solr pull out 10000 or 5000 records for each
>> page(Divided to 100 times or 200 times queries) . Then just print out these
>> records to the client browser.
>> I am not sure how the exception was generated?
>> Is my client program(the Servlet program)  out of memory?or Connect timeout
>> for some reason?
>> This exception doesn't always happen. Sometimes it works well even I query
>> 10000 records each time and works for many times , but sometimes it crashes
>> only 5000 records without an explicit reason.
>> You suggestion is great! But the implementation is a little complicated for
>> us.
>> Is Lucene better than Solr for this requirement? But the paging in Lucene
>> seems not very intuitively.
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Fail-to-huge-collection-extraction-tp4003559p4006450.html
>> Sent from the Solr - User mailing list archive at Nabble.com.