lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SOLR-846) Out Of memory doing delta import with fetch size set to -1
Date Fri, 07 Sep 2012 21:40:07 GMT

     [ https://issues.apache.org/jira/browse/SOLR-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hoss Man resolved SOLR-846.
---------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 4.0)
                   1.4

this issue was listed as "fixed" in Solr 1.4's CHANGES.txt.

in light of that, i'm resolving, and any future work to improve things can be done in a new
issue (since a new issue id would be neccessary to properly track it in CHANGES.txt anyway)
                
> Out Of memory doing delta import with fetch size set to -1
> ----------------------------------------------------------
>
>                 Key: SOLR-846
>                 URL: https://issues.apache.org/jira/browse/SOLR-846
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>         Environment: Linux 2.6.18-92.1.13.el5xen, mysql 5.0
>            Reporter: Ricky Leung
>             Fix For: 1.4
>
>         Attachments: SOLR-846.patch
>
>
> Database has about 3 million records.  Doing full-import there is no problem.  However,
when a large number of changes occurred 2558057, delta-import throws OutOfMemory error after
1288338 documents processed.  The stack trace is below
> Exception in thread "Thread-3" java.lang.OutOfMemoryError: Java heap space
> 	at org.tartarus.snowball.ext.EnglishStemmer.<init>(EnglishStemmer.java:4
> 9)
> 	at org.apache.solr.analysis.EnglishPorterFilter.<init>(EnglishPorterFilt
> erFactory.java:83)
> 	at org.apache.solr.analysis.EnglishPorterFilterFactory.create(EnglishPor
> terFilterFactory.java:66)
> 	at org.apache.solr.analysis.EnglishPorterFilterFactory.create(EnglishPor
> terFilterFactory.java:35)
> 	at org.apache.solr.analysis.TokenizerChain.tokenStream(TokenizerChain.ja
> va:48)
> 	at org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer.tokenStream(Inde
> xSchema.java:348)
> 	at org.apache.lucene.analysis.Analyzer.reusableTokenStream(Analyzer.java
> :44)
> 	at org.apache.lucene.index.DocInverterPerField.processFields(DocInverter
> PerField.java:117)
> 	at org.apache.lucene.index.DocFieldConsumersPerField.processFields(DocFi
> eldConsumersPerField.java:36)
> 	at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(Do
> cFieldProcessorPerThread.java:234)
> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWrite
> r.java:765)
> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWrite
> r.java:748)
> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2
> 118)
> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2
> 095)
> 	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandle
> r2.java:232)
> 	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpd
> ateProcessorFactory.java:59)
> 	at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:
> 69)
> 	at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImp
> ortHandler.java:288)
> 	at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
> r.java:319)
> 	at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java
> :211)
> 	at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
> :133)
> 	at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImp
> orter.java:359)
> 	at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
> ava:388)
> 	at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
> va:377)
> dataSource in data-config.xml has been with the batchSize of "-1".
>     <dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql://host/dbname" 
> user="*" password="*" batchSize="-1"/>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message