Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of pranny@gmail.com designates
 74.125.82.48 as permitted sender)
MIME-Version: 1.0
From: Pranav Prakash <pranny@gmail.com>
Date: Tue, 4 Oct 2011 18:47:34 +0530
Message-ID: 
 <CAM6QAw3icPxshJg6Q+N2Rms51nnUM1H0u+-zaoHqwyStWzUNug@mail.gmail.com>
Subject: How to achieve Indexing @ 270GiB/hr
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00504502cd8550ed9504ae78eb16

--00504502cd8550ed9504ae78eb16
Content-Type: text/plain; charset=ISO-8859-1

Greetings,

While going through the article 265% indexing speedup with Lucene's
concurrent flushing<http://java.dzone.com/news/265-indexing-speedup-lucenes?mz=33057-solr_lucene>
I
was stunned by the endless possibilities in which Indexing speed could be
increased.

I'd like to take inputs from everyone over here as to how to achieve this
speed. As far as I understand there are two broad ways of feeding data to
Solr -

   1. Using DataImportHandler
   2. Using HTTP to POST docs to Solr.

The speeds at which the article describes indexing seems kinda too much to
expect using the second approach. Or is it possible using multiple instances
feeding docs to Solr?

My current setup does the following -

   1. Execute SQL queries to create database of documents that needs to be
   fed.
   2. Go through the columns one by one, and create XMLs for them and send
   it over to Solr in batches of max 500 docs.


Even if using DataImportHandler what are the ways this could be optimized?
If I am able to solve the problem of indexing data in our current setup, my
life would become a lot easier.


*Pranav Prakash*

"temet nosce"

Twitter <http://twitter.com/pranavprakash> | Blog <http://blog.myblive.com> |
Google <http://www.google.com/profiles/pranny>

--00504502cd8550ed9504ae78eb16--