Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID;
  b=ni9j+9ggsIcy1oA6R7dUoOZNGygjkKoLux/ivVH4LPp07PrXUbEvqLjDaxY0hxRQIuDr9ZfGjuH5LLjDTtsj2zj7THcDHqoAtNM6s4XLDPnOJ76gnM0KeCHoSLbgU/4aFVD8Zj39jk/tQUsiCSYlzktC7+g2DTDrZEMWVjyFJms=;
Date: Wed, 17 Sep 2008 23:47:18 -0700 (PDT)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: Exception while doing sorting
To: java-user@lucene.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Message-ID: <661170.73589.qm@web50312.mail.re2.yahoo.com>

If your index is increasing in size so fast, you should start thinking about sharding your index (breaking it into multiple smaller indices that each fits on its server) and searching across them (aka distributed search).

Yes, Lucene can handle millions of records if run on adequate hardware and if used correctly.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


----- Original Message ----
> From: Ganesh - yahoo <emailgane@yahoo.co.in>
> To: java-user@lucene.apache.org
> Sent: Thursday, September 18, 2008 12:53:19 AM
> Subject: Re: Exception while doing sorting
> 
> My index is growing by 1 million records per day. How much memory do i need 
> to increase.
> 
> What kind of sorting algorithm being used in Lucene. Is this efficient 
> enough to handle millions of records.
> 
> Whether we could do sorting using our own algorithm?
> 
> Regards
> Ganesh
> 
> ----- Original Message ----- 
> From: "Fuad Efendi" 
> To: 
> Sent: Wednesday, September 17, 2008 7:28 PM
> Subject: Re: Exception while doing sorting
> 
> 
> Increase memory.
> 
> Lucene uses FieldCache for sorting on non-tokenized field and tries to
> maintain fields from all your 4 millions documents, even if you need
> to sort only 4000 docs.
> ==============
> http://www.tokenizer.org/bot.html
> 
> 
> Quoting Ganesh - yahoo :
> 
> > Hello all,
> >
> > I am have indexed more than 4 million documents. My query fetches
> > 300,000 hits. If i perform sorting on any field, then tomcat reports
> > out of memory exception.
> > Sometimes the query results may be around 1000, but sorting on any
> > field might take more than 30 - 50 secs.
> >
> > I don't know what's going wrong.
> >
> > My index searcher is static object and it is getting refreshed every
> > minute. JSP pages directly calls the index searcher object and performs
> > search.
> >
> > Regards
> > Ganesh
> >
> >
> > Send instant messages to your online friends
> > http://in.messenger.yahoo.com
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> Send instant messages to your online friends http://in.messenger.yahoo.com 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org