lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avlesh Singh <avl...@gmail.com>
Subject Re: Queries regarding a "ParallelDataImportHandler"
Date Sun, 02 Aug 2009 16:09:38 GMT
>
> There can be a batch command (which) will take in multiple commands in one
> http request.
>
You seem to be obsessed with this approach, Noble.
Solr-1093<http://issues.apache.org/jira/browse/SOLR-1093>also echoes
the same sentiments :)
I personally find this approach a bit restrictive and difficult to adapt to.
IMHO, it is better handled as a configuration. i.e. user tells us how the
single task can be "batched" (or 'sliced', as you call it) while configuring
the Parallel(or, MultiThreaded) DIH inside solrconfig.

As an example, for non-jdbc data sources where batching might be difficult
to achieve in an abstract way, the user might choose to configure different
data-config.xml's (for different DIH instances) altogether.

Cheers
Avlesh

2009/8/2 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@corp.aol.com>

> On Sun, Aug 2, 2009 at 8:56 PM, Avlesh Singh<avlesh@gmail.com> wrote:
> > I have one more question w.r.t the MultiThreaded DIH - What would be the
> > logic behind distributing tasks to thread?
> >
> > I am sorry to have not mentioned this earlier - In my case, I take a
> "count
> > query" parameter as an configuration element. Based on this count and the
> > maxNumberOfDIHInstances, task assignment scheduling is done by
> "injecting"
> > limit and offset values in the import query for each DIH instance.
> > And this is, one of the reasons, why I call it a
> ParallelDataImportHandler.
> There can be a batch command will take in multiple commands in one
> http request. so it will be like invoking multiple DIH instances and
> the user will have to find ways to split up the whole task into
> multiple 'slices'. DIH in turn would fire up multiple threads and once
> all the threads are returned it should issue a commit
>
> this is a very dumb implementation but is a very easy path.
> >
> > Cheers
> > Avlesh
> >
> > On Sun, Aug 2, 2009 at 8:39 PM, Avlesh Singh <avlesh@gmail.com> wrote:
> >
> >> run the add() calls to Solr in a dedicated thread
> >>
> >> Makes absolute sense. This would actually mean, DIH sits on top of all
> the
> >> add/update operations making it easier to implement a multi-threaded
> DIH.
> >>
> >> I would create a JIRA issue, right away.
> >> However, I would still love to see responses to my problems due to
> >> limitations in 1.3
> >>
> >> Cheers
> >> Avlesh
> >>
> >> 2009/8/2 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@corp.aol.com>
> >>
> >> a multithreaded DIH is in my top priority list. There are muliple
> >>> approaches
> >>>
> >>> 1) create multiple instances of dataImporter instances in the same DIH
> >>> instance and run them in parallel and commit when all of them are done
> >>> 2) run the add() calls to Solr in a dedicated thread
> >>> 3) make DIH automatically multithreaded . This is much harder to
> >>> implement.
> >>>
> >>> but a and #1 and #2 can be implemented with ease. It does not have to
> >>> be aother implementation called ParallelDataImportHandler. I believe
> >>> it can be done in DIH itself
> >>>
> >>> you may not need to create a project in google code. you can open a
> >>> JIRA issue and start posting patches and we can put it back into Solr.
> >>>
> >>> .
> >>>
> >>> On Sun, Aug 2, 2009 at 7:33 PM, Avlesh Singh<avlesh@gmail.com> wrote:
> >>> > In my quest to improve indexing time (in a multi-core environment),
I
> >>> tried
> >>> > writing a Solr RequestHandler called ParallelDataImportHandler.
> >>> > I had a few lame questions to begin with, which Noble and Shalin
> >>> answered
> >>> > here -
> >>> >
> >>>
> http://www.lucidimagination.com/search/document/22b7371c063fdb06/using_dih_for_parallel_indexing
> >>> >
> >>> > As the name suggests, the handler, when invoked, tries to execute
> >>> multiple
> >>> > DIH instances on the same core in parallel. Of-course the catch here
> is
> >>> > that, only those data-sources, that can be batched can benifit from
> this
> >>> > handler. In my case, I am writing this for import from a MySQL
> database.
> >>> So,
> >>> > I have a single data-config.xml, in which the query has to add
> >>> placeholders
> >>> > for "limit" and "offset". Each DIH instance uses the same data-config
> >>> file,
> >>> > and replaces its own values for the limit and offset (which is in
> fact
> >>> > supplied by the parent ParallelDataImportHandler).
> >>> >
> >>> > I am achieving this by making my handler SolrCoreAware, and creating
> >>> > maxNumberOfDIHInstances (configurable) in the inform method. These
> >>> instances
> >>> > are then initialized and  registered with the core. Whenever a
> request
> >>> comes
> >>> > in, the ParallelDataImportHandler delegates the task to these
> instances,
> >>> > schedules the remainder and aggregates responses from each of these
> >>> > instances to return back to the user.
> >>> >
> >>> > Thankfully, all of these worked, and preliminary benchmarking with
> >>> 5million
> >>> > records indicated 50% decrease in re-indexing time. Moreover, all my
> >>> cores
> >>> > (Solr in my case is hosted on a quad-core machine), indicated above
> 70%
> >>> CPU
> >>> > utilization. All that I could have asked for!
> >>> >
> >>> > With respect to this whole thing, I have a few questions -
> >>> >
> >>> >   1. Is something similar available out of the box?
> >>> >   2. Is the idea flawed? Is the approach fundamentally correct?
> >>> >   3. I am using Solr 1.3. DIH did not have "EventListeners" in the
> stone
> >>> >   age. I need to know, if a DIH instance is done with its task
> (mostly
> >>> the
> >>> >   "commit") operation. I could not figure a clean way out. As a hack,
> I
> >>> keep
> >>> >   pinging the DIH instances with command=status at regular intervals
> (in
> >>> a
> >>> >   separate thread), to figure out if it is free to be assigned some
> >>> task. With
> >>> >   works, but obviously with an overhead of unnessecary wasted CPU
> >>> cycles. Is
> >>> >   there a better approach?
> >>> >   4. I can better the time taken, even further if there was a way for
> me
> >>> to
> >>> >   tell a DIH instance not to open a new IndexSearcher. In the current
> >>> scheme
> >>> >   of things, as soon as one DIH instance is done committing, a new
> >>> searcher is
> >>> >   opened. This is blocking for other DIH instances (which were
> active)
> >>> and
> >>> >   they cannot continue without the searcher being initialized. Is
> there
> >>> a way
> >>> >   I can implement, single commit once all these DIH instances are
> done
> >>> with
> >>> >   their tasks? I tried each DIH instance with a commit=false without
> >>> luck.
> >>> >   5. Can this implementation be extended to support other
> data-sources
> >>> >   supported in DIH (HTTP, File, URL etc)?
> >>> >   6. If the utility is worth it, can I host this on Google code as
an
> >>> open
> >>> >   source contrib?
> >>> >
> >>> > Any help will be deeply acknowledged and appreciated. While
> suggesting,
> >>> > please don't forget that I am using Solr 1.3. If it all goes well,
I
> >>> don't
> >>> > mind writing one for Solr 1.4.
> >>> >
> >>> > Cheers
> >>> > Avlesh
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> -----------------------------------------------------
> >>> Noble Paul | Principal Engineer| AOL | http://aol.com
> >>>
> >>
> >>
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message