lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avlesh Singh <>
Subject Queries regarding a "ParallelDataImportHandler"
Date Sun, 02 Aug 2009 14:03:04 GMT
In my quest to improve indexing time (in a multi-core environment), I tried
writing a Solr RequestHandler called ParallelDataImportHandler.
I had a few lame questions to begin with, which Noble and Shalin answered
here -

As the name suggests, the handler, when invoked, tries to execute multiple
DIH instances on the same core in parallel. Of-course the catch here is
that, only those data-sources, that can be batched can benifit from this
handler. In my case, I am writing this for import from a MySQL database. So,
I have a single data-config.xml, in which the query has to add placeholders
for "limit" and "offset". Each DIH instance uses the same data-config file,
and replaces its own values for the limit and offset (which is in fact
supplied by the parent ParallelDataImportHandler).

I am achieving this by making my handler SolrCoreAware, and creating
maxNumberOfDIHInstances (configurable) in the inform method. These instances
are then initialized and  registered with the core. Whenever a request comes
in, the ParallelDataImportHandler delegates the task to these instances,
schedules the remainder and aggregates responses from each of these
instances to return back to the user.

Thankfully, all of these worked, and preliminary benchmarking with 5million
records indicated 50% decrease in re-indexing time. Moreover, all my cores
(Solr in my case is hosted on a quad-core machine), indicated above 70% CPU
utilization. All that I could have asked for!

With respect to this whole thing, I have a few questions -

   1. Is something similar available out of the box?
   2. Is the idea flawed? Is the approach fundamentally correct?
   3. I am using Solr 1.3. DIH did not have "EventListeners" in the stone
   age. I need to know, if a DIH instance is done with its task (mostly the
   "commit") operation. I could not figure a clean way out. As a hack, I keep
   pinging the DIH instances with command=status at regular intervals (in a
   separate thread), to figure out if it is free to be assigned some task. With
   works, but obviously with an overhead of unnessecary wasted CPU cycles. Is
   there a better approach?
   4. I can better the time taken, even further if there was a way for me to
   tell a DIH instance not to open a new IndexSearcher. In the current scheme
   of things, as soon as one DIH instance is done committing, a new searcher is
   opened. This is blocking for other DIH instances (which were active) and
   they cannot continue without the searcher being initialized. Is there a way
   I can implement, single commit once all these DIH instances are done with
   their tasks? I tried each DIH instance with a commit=false without luck.
   5. Can this implementation be extended to support other data-sources
   supported in DIH (HTTP, File, URL etc)?
   6. If the utility is worth it, can I host this on Google code as an open
   source contrib?

Any help will be deeply acknowledged and appreciated. While suggesting,
please don't forget that I am using Solr 1.3. If it all goes well, I don't
mind writing one for Solr 1.4.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message