Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
MIME-Version: 1.0
From: David Hastings <hastings.recursive@gmail.com>
Date: Fri, 10 Feb 2017 16:55:29 -0500
Message-ID: <CANGA2_PV_TZQcQe-UFW7Wg7ZBiute2BKBviHcOeqGBbp5tBbrg@mail.gmail.com>
Subject: commongrams
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001a113deb90ff99bd0548342722
archived-at: Fri, 10 Feb 2017 21:55:41 -0000

--001a113deb90ff99bd0548342722
Content-Type: text/plain; charset=UTF-8

Hey All,
I followed an old blog post about  implementing the common grams, and used
the 400 most popular words file on a subset of my data.  original index
size was 33gb with 2.2 million documents, using the 400, it grep to 96gb.
I scaled it down to the 100 most common words and got to about 76gb, but
with a cold phrase search going from 4 seconds at 400 words to 6 with 100.
 this will not really scale well, as the base index that this is a subset
of right now has 22 million documents and sits around 360 gb.  at this
rate, it would be around a TB index size.  is there a common
hardware/software configuration to handle TB size indexes?
thanks,
DH

--001a113deb90ff99bd0548342722--