lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Adams" <jasolru...@gmail.com>
Subject Re: how large can the index be?
Date Wed, 07 Jan 2009 02:57:17 GMT
Why is NFS mounting such a bad idea? Some solutions for high available disks
suggest that you DO mount the disks NFS to the boxes that need the data.

On Mon, Dec 29, 2008 at 7:42 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:

> What you have below is not really what we call "Distributed Search", but
> more of "Query Load Balancing".  Yes, the diagram below will work IF a
> single Solr box (A or B) can really handle a full 50M doc index.  Of course
> "handle" can be fuzzy.  That is, you could have a large index on a Solr box
> and it will "handle" it - nothing will crash, nothing will die, it's just
> that it may not be able to handle it "well enough" - that is, the queries
> may take longer than you'd like.
>
> NFS mounting an index directory is a separate story and very often a bad
> idea, again because of performance.
>
>
> Otis --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Antonio Eggberg <antonio_eggberg@yahoo.se>
> > To: solr-user@lucene.apache.org
> > Sent: Monday, December 29, 2008 4:19:23 PM
> > Subject: Re: how large can the index be?
> >
> > Thanks you very much for your answer.
> >
> > I was afraid of that the each document has about 20 fields.. As you
> pointed out
> > it will slow down. Anyway I am thinking is it not possible to do the
> following:
> >
> > Load Balancer
> >      |
> > Solr A, Solr B, ...
> >      |
> >   one index
> >
> > So I send 50% query to Solr A, 50% to Solr B and so forth.. is this not
> good?
> > Also to add The index will be like a mounted drive to the solr boxes...
> On the
> > above do I really need to worry about Solr Master, Solr Slave? It
> probably solve
> > my load but I think query speed will be slow...
> >
> > Just curious anyone using distributed search in production?
> >
> > Cheers
> >
> >
> >
> > --- Den mån 2008-12-29 skrev Otis Gospodnetic :
> >
> > > Från: Otis Gospodnetic
> > > Ämne: Re: how large can the index be?
> > > Till: solr-user@lucene.apache.org
> > > Datum: måndag 29 december 2008 21.53
> > > Hi Antonio,
> > >
> > > Besides thinking in terms of documents, you also need to
> > > think in terms of index size on the file system vs. the
> > > amount of RAM your search application/server can use.  50M
> > > documents may be doable on a single server if those
> > > documents are not too large and you have sufficient RAM.  It
> > > gets even better if your index doesn't change very often
> > > and if you can get decent hit ratios on the various Solr
> > > caches.
> > >
> > > If you are indexing largish documents, or even something as
> > > small as an average web page, 50M docs may be too much on a
> > > "commodity box" (say dual core 8 GB RAM box)
> > >
> > > Otis
> > > --
> > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > >
> > >
> > >
> > > ----- Original Message ----
> > > > From: Antonio Eggberg
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Monday, December 29, 2008 3:41:48 PM
> > > > Subject: how large can the index be?
> > > >
> > > > Hi,
> > > >
> > > > We are running successfully a solr index of 3 million
> > > docs. I have just been
> > > > informed that our index size will increase to 50
> > > million. I been going through
> > > > the doc
> > > >
> > > > http://wiki.apache.org/solr/DistributedSearch
> > > >
> > > > Seems like we will loose out on the date facet and
> > > some more other stuff that we
> > > > use. which is important to us. So far we been using 1
> > > index and 1 machine.
> > > >
> > > > Can I still stick with my 1 index but have many query
> > > servers? We don't update
> > > > our index so often this are rather static data. Over
> > > the past year we have
> > > > updated the index data a total of 3 times and about
> > > 300 records :)
> > > >
> > > > Can someone provide some idea how/what should I do to
> > > deal with new datasets?.
> > > >
> > > > Thanks for your help.
> > > >
> > > >
> > > >
> > > __________________________________________________________
> > > > Går det långsamt? Skaffa dig en snabbare
> > > bredbandsuppkoppling.
> > > > Sök och jämför priser hos Kelkoo.
> > > >
> > > http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325
> >
> >
> >       __________________________________________________________
> > Låna pengar utan säkerhet. Jämför vilkor online hos Kelkoo.
> >
> http://www.kelkoo.se/c-100390123-lan-utan-sakerhet.html?partnerId=96915014
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message