lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-3585) processing updates in multiple threads
Date Tue, 10 Jul 2012 05:42:34 GMT

    [ https://issues.apache.org/jira/browse/SOLR-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410065#comment-13410065
] 

David Smiley commented on SOLR-3585:
------------------------------------

I wasn't clear at all, sorry.  Let me try again:

I like your patch because *it puts indexing concurrency control into the server where it belongs*.
 Presently, to take advantage of a indexing to a beefy Solr server as fast as possible, it
is basically up to the client to do it in multiple concurrent connections.  StreamingUpdateSolrServer
can do this (and the DIH can too although it's deprecated) but if you're using something else
(e.g. ruby) then you would have to code it yourself to load documents faster, which is challenging
-- concurrency is hard.  And furthermore, the client shouldn't be making assumptions as to
how many indexing threads Solr can handle -- that should be in Solr's configuration.

I think it would be nicer if this concurrency behavior you coded as an UpdateRequestProcessor
was instead something internal to Solr, somewhere in-between where an UpdateRequestHandler
receives its data and hands it off to an URP chain.  This currently happens in ContentStreamHandlerBase.
 Maybe UpdateRequestProcessorChain.createProcessor() would do it; I'm not sure -- there isn't
a perfectly clean place for it based on Solr's architecture, unfortunately.
                
> processing updates in multiple threads
> --------------------------------------
>
>                 Key: SOLR-3585
>                 URL: https://issues.apache.org/jira/browse/SOLR-3585
>             Project: Solr
>          Issue Type: Improvement
>          Components: update
>    Affects Versions: 4.0
>            Reporter: Mikhail Khludnev
>            Priority: Minor
>         Attachments: SOLR-3585.patch, SOLR-3585.patch, multithreadupd.patch, report.tar.gz
>
>
> Hello,
> I'd like to contribute update processor which forks many threads which concurrently process
the stream of commands. It may be beneficial for users who streams many docs through single
request. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message