Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID;
  b=Ug1Z8zRH5lIvNQD3W3SB5p/CpJJhX4xRGOCIO9ArSYXgm+eAxuQTvXADR2wHTUl0rrlZijXQtOWRl+0fdY84h8Y06WiqYnIT6d06pG48p7PU+SJ8LQc1YJcVB5vZ1osg/1fUBPLYKhBthPORlaoyRHFoqLRypf9wekU1OKBVKlg=;
References: <490F356B.1080806@umich.edu> <49131672.1080301@umich.edu>
Date: Thu, 6 Nov 2008 08:42:34 -0800 (PST)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: Huge increase in index size adding just 2 fields
To: solr-user@lucene.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Message-ID: <897649.4898.qm@web50306.mail.re2.yahoo.com>

I'll make a very wild guess and say that it's possible for this to happen if your dates are very granular (down to milliseconds).  All of a sudden you probably got 500,000 new terms there.  Wild guess.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


----- Original Message ----
> From: Phillip Farber <pfarber@umich.edu>
> To: solr-user@lucene.apache.org
> Sent: Thursday, November 6, 2008 11:08:18 AM
> Subject: Re: Huge increase in index size adding just 2 fields
> 
> May I ask again whether a index size increase from 120GB to 166GB is expected 
> simply by adding a stored date and a stored repeating string field if length 
> perhaps 20 and roughly 2 values per doc for 500,000 on average?  The doc is a 
> large body of OCR and the position index dominates due to the large number of 
> terms.
> 
> Thanks,
> 
> Phil
> 
> 
> Phillip Farber wrote:
> > 
> > Hi,
> > 
> > We're indexing a lot of dirty OCR. So the index is really huge due to the size 
> of the position file.  We still get ok response time though with a median of 
> 100ms.  Phrase queries are a different matter obviously.  But we're seeing some 
> really large increases in index size as we add a couple of fields that do not 
> make sense.
> > 
> > Our 500,000 document index is 120G. It's simple schema is:
> > 
> > 
> > 
> > 
> > 
> > 
> required="true"/>
> > 
> > We added the following 2 fields to the above schema as follows:
> > 
> > 
> > 
> multiValued="true"/>
> > 
> > where the "hlb" field consists of not more than 3-4 strings such as "Social 
> Sicence"/
> > 
> > Our 500,000 document index size increased to 166G!  This seems completely 
> wrong.  Looking at the directory listings for each case it appears every one of 
> the files grew in size.
> > 
> > How can this be?
> > 
> > Phil
> > 
> > ===
> > 
> > 120G index:
> > 
> > -rw-r--r--  1 tomcat admin     81023261 Sep 24 06:00 _fj.fdt
> > -rw-r--r--  1 tomcat admin      4000072 Sep 24 06:00 _fj.fdx
> > -rw-r--r--  1 tomcat admin           33 Sep 24 06:00 _fj.fnm
> > -rw-r--r--  1 tomcat admin  14069125169 Sep 24 06:16 _fj.frq
> > -rw-r--r--  1 tomcat admin      1500031 Sep 24 06:16 _fj.nrm
> > -rw-r--r--  1 tomcat admin 109247382360 Sep 24 08:25 _fj.prx
> > -rw-r--r--  1 tomcat admin     58677668 Sep 24 08:25 _fj.tii
> > -rw-r--r--  1 tomcat admin   4319853217 Sep 24 08:32 _fj.tis
> > -rw-r--r--  1 tomcat admin           42 Sep 24 08:32 segments_fo
> > -rw-r--r--  1 tomcat admin           20 Sep 24 08:32 segments.gen
> > 
> > 166G index (+ 2 fields)
> > 
> > -rw-r--r-- 1 tomcat admin    113530692 Oct 21 10:42 _fh.fdt
> > -rw-r--r-- 1 tomcat admin      3960256 Oct 21 10:42 _fh.fdx
> > -rw-r--r-- 1 tomcat admin           44 Oct 21 10:42 _fh.fnm
> > -rw-r--r-- 1 tomcat admin  15242830112 Oct 21 12:58 _fh.frq
> > -rw-r--r-- 1 tomcat admin      1485100 Oct 21 12:58 _fh.nrm
> > -rw-r--r-- 1 tomcat admin 117927610810 Oct 21 12:58 _fh.prx
> > -rw-r--r-- 1 tomcat admin     72760439 Oct 21 12:58 _fh.tii
> > -rw-r--r-- 1 tomcat admin   5337669551 Oct 21 12:58 _fh.tis
> > -rw-r--r-- 1 tomcat admin           42 Oct 21 12:58 segments_fk
> > -rw-r--r-- 1 tomcat admin           20 Oct 21 12:58 segments.gen