lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Khludnev (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-3585) processing updates in multiple threads
Date Tue, 03 Jul 2012 21:24:33 GMT

     [ https://issues.apache.org/jira/browse/SOLR-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mikhail Khludnev updated SOLR-3585:
-----------------------------------

    Attachment: SOLR-3585.patch

Folks, why are you so skeptic? Don't you need utilize your boxes to get indexing happen faster?

The also potential usage which I see is returning multithreading to DIH back by keeping the
old core single threaded, then putting this multithread consumer behind and pushing into from
improved datasources which will have buffer and separate threads. 

ok. here is java doc from the recent patch. most update is correct and comprehensive halting
implementation.

/**
 * Forks incoming adds into multiple threads. 
 * Instead of regular chaining it chains on it's own with chain specified in 
 * "backing.chain" update parameter. 
 * supports two solrconfig.xml parameters <ul>
 * <li>
 * "bufferSize" length of the buffer for add commands
 * </li>
 * <li>"pipesNumber" number of parallel threads for handling updates</li>
 * </ul>
 * it spawns own cached thread pool, and shutdown it on core closing.
 * AddUpdate commands are queued for background execution by spawned processors.
 * All other commands awaits until all queued tasks are completed, and then are 
 * handled in the calling thread (request).  
 * if one of the update processors threads (aka pipes) catches exception, all other pipes
 * are stopped synchronously, but not in the same moment, and the root cause is propagated

 * into request thread. 
 * for queuing spin-delay pattern is used to prevent dead lock when buffer is full, and
 * queuing thread is blocked, but buffer can't be processed due to dead pipes.       
 * */
                
> processing updates in multiple threads
> --------------------------------------
>
>                 Key: SOLR-3585
>                 URL: https://issues.apache.org/jira/browse/SOLR-3585
>             Project: Solr
>          Issue Type: Improvement
>          Components: update
>    Affects Versions: 4.0
>            Reporter: Mikhail Khludnev
>            Priority: Minor
>         Attachments: SOLR-3585.patch, multithreadupd.patch
>
>
> Hello,
> I'd like to contribute update processor which forks many threads which concurrently process
the stream of commands. It may be beneficial for users who streams many docs through single
request. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message