lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitry Kan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-3585) processing updates in multiple threads
Date Thu, 30 Aug 2012 19:59:07 GMT

    [ https://issues.apache.org/jira/browse/SOLR-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445246#comment-13445246
] 

Dmitry Kan commented on SOLR-3585:
----------------------------------

Summary:

1/2/4/8 threads

There was a gain for 2 threads, after that increasing amount of threads didn't matter for
the indexing speed (again, can be too little data, too slow machine vs server)

URL:

http://localhost:8983/solr/update?commit=true&separator=%09&escape=\&update.chain=threads&backing.chain=logrun&stream.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example\data\book_edition.tsv&stream.contentType=text/csv;charset=utf-8

Intel(R) Core2 Duo CPU T6600 @ 2.20GHz
RAM: 4 GB
OS: Windows 7 64 bit

PC was moderately used during the indexing (Internet surfing mostly)

Solr started with:
java -Xmx512M -Xms512M -jar start.jar

Stats and Log extract:

-------------------
one thread
-------------------

565576 milliseconds (9.43 seconds)
size of data/index: 1.61 GB

30.08.2012 22:34:10 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/update params={backing.chain=logrun&commi
t=true&stream.contentType=text/csv;charset%3Dutf-8&separator=%09&escape=\&stream
.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example
\data\book_edition.tsv&update.chain=threads} {add=[/m/0g9nk5p, /m/0g9rf0q, /m/0g
j6_r3, /m/0gj702y, /m/0gk99b7, /m/0g461_s, /m/0g4thbr, /m/0g4vp__, /m/0gkgw7x, /
m/0gb390f, ... (3401498 adds)]} 0 565576

-------------------
two threads
-------------------

400085 milliseconds (6.67 seconds)
size of data/index: 916MB

30.08.2012 22:09:16 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1

30.08.2012 22:15:56 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/update params={backing.chain=logrun&commit=true&stream.contentType=text/csv;charset%3Dutf-8&separator=%09&escape=\&stream.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example\data\book_edition.tsv&update.chain=threads}
{add=[/m/0g9nk5p, /m/0gj6_r3, /m/0gkgw7x, /m/0g9_qhd, /m/0g9_r1t, /m/0g9jxyt, /m/0g4wdtq,
/m/0d0s9y1, /m/0d9pb_v, /m/0d0tfz7, ... (1838414 adds)]} 0 400085
30.08.2012 22:15:56 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/update params={backing.chain=logrun&commit=true&stream.contentType=text/csv;charset%3Dutf-8&separator=%09&escape=\&stream.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example\data\book_edition.tsv&update.chain=threads}
{add=[/m/0g9rf0q, /m/0gj702y, /m/0gk99b7, /m/0g461_s, /m/0g4thbr, /m/0g4vp__, /m/0gb390f,
/m/0gb34pf, /m/0h8fm59, /m/0g99vfk, ... (1563084 adds)]} 0 400085


-------------------
four threads
-------------------

423969 milliseconds (7.07 seconds)
size of data/index: 915 MB

30.08.2012 21:52:03 org.apache.solr.core.SolrDeletionPolicy updateCommits

INFO: [collection1] webapp=/solr path=/update params={backing.chain=logrun&commit=true&stream.contentType=text/csv;charset%3Dutf-8&separator=%09&escape=\&stream.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example\data\book_edition.tsv&update.chain=threads}
{add=[/m/0g9nk5p, /m/0dgjnsn, /m/0d0s539, /m/0d0t8b3, /m/0d9n2sg, /m/0d0s18j, /m/07n7lbm,
/m/07n7mh6, /m/07n7mq0, /m/07n7n_d, ... (844367 adds)]} 0 r
30.08.2012 21:59:07 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0gj702y, /m/0gk99b7, /m/0gkgw7x, /m/0gb390f, /m/0g9_qhd, /m/0h2ymt3,
/m/0g4wdtq, /m/0d0s9y1, /m/0d0tfz7, /m/0d0tdf1, ... (815450 adds)]} 0 423969
30.08.2012 21:59:07 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0g9rf0q, /m/0g461_s, /m/0g4thbr, /m/0g4vp__, /m/0gb34pf, /m/0h8fm59,
/m/0g99vfk, /m/0g9_r1t, /m/0g9jxyt, /m/0ghc2b5, ... (836534 adds)]} 0 423969
30.08.2012 21:59:07 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/update params={backing.chain=logrun&commit=true&stream.contentType=text/csv;charset%3Dutf-8&separator=%09&escape=\&stream.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example\data\book_edition.tsv&update.chain=threads}
{add=[/m/0gj6_r3, /m/0d0sfq_, /m/0d9mhx1, /m/07tc6lf, /m/07tc75v, /m/07tc7jq, /m/07tc8kz,
/m/07tc8wr, /m/07tc_cn, /m/07tc_fl, ... (905147 adds)]} 0 423969

-------------------
eight threads
-------------------

431710 milliseconds (7.20 seconds)
size of data/index: 1.00 GB


30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/update params={backing.chain=logrun&commit=true&stream.contentType=text/csv;charset%3Dutf-8&separator=%09&escape=\&stream
.file=d:\Projects\information_retrieval\solr\apache-solr-4.0.0-BETA\solr\example\data\book_edition.tsv&update.chain=threads}
{add=[/m/0gk99b7, /m/0d0vb6s, /m/07t8mw8, /m/07t8pvt, /m/07t8ygz, /m/07t8yr6, /m/07t904r,
/m/07t90cq, /m/07t91l8, /m/07t91yl, ... (369913 adds)]} 0 431710
30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0g9rf0q, /m/0dgh_yx, /m/0d0t2jm, /m/0d0tkmp, /m/0d0tdjk, /m/0d0s4t_,
/m/0d9mcxz, /m/0d0rzm4, /m/0d9mjyb, /m/0d9n806, ... (480781 adds)]} 0 431710
30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0g4thbr, /m/07tct8c, /m/07tcwx7, /m/07tcxx_, /m/07tczs9, /m/07td4wf,
/m/07td5z1, /m/07tp81q, /m/07tpf2f, /m/07tphh7, ... (331760 adds)]} 0 431710
30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0gj6_r3, /m/0dghv3t, /m/0dghm0z, /m/0dghgk7, /m/0d0s99z, /m/0d9mkv1,
/m/0dgj61b, /m/0d9mkns, /m/0dgj3st, /m/0d0v7p7, ... (368899 adds)]} 0 431710
30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0g9nk5p, /m/0dj5yc2, /m/0dghh9b, /m/0dghhyt, /m/0dghkcc, /m/0dgjcyy,
/m/0d0sfq_, /m/0d9nbgx, /m/0d0s51w, /m/0d9mby3, ... (374343 adds)]} 0 431710
30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0g4vp__, /m/0g99vfk, /m/0h2ymt3, /m/0dghty2, /m/0d9pb_v, /m/0d0st88,
/m/0dghh0h, /m/0dghh_g, /m/0d9nmdk, /m/0dghspk, ... (504140 adds)]} 0 431710
30.08.2012 22:47:43 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0gj702y, /m/0gkgw7x, /m/0gb390f, /m/0gb34pf, /m/0h8fm59, /m/0g9_qhd,
/m/0g9_r1t, /m/0g9jxyt, /m/0ghc2b5, /m/0g4wdtq, ... (450096 adds)]} 0 431710
30.08.2012 22:47:45 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] {add=[/m/0g461_s, /m/07tptf3, /m/07tzgzv, /m/07v1bqt, /m/07v1d9p, /m/07v1fbh,
/m/07v1jgm, /m/07v663l, /m/07v6fbt, /m/07v6gt5, ... (521566 adds)]} 0 431710
                
> processing updates in multiple threads
> --------------------------------------
>
>                 Key: SOLR-3585
>                 URL: https://issues.apache.org/jira/browse/SOLR-3585
>             Project: Solr
>          Issue Type: Improvement
>          Components: update
>    Affects Versions: 4.0-ALPHA
>            Reporter: Mikhail Khludnev
>            Priority: Minor
>         Attachments: multithreadupd.patch, report.tar.gz, SOLR-3585.patch, SOLR-3585.patch
>
>
> Hello,
> I'd like to contribute update processor which forks many threads which concurrently process
the stream of commands. It may be beneficial for users who streams many docs through single
request. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message