lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruben Jimenez (JIRA)" <j...@apache.org>
Subject [jira] Commented: (SOLR-857) Memory Leak during the indexing of large xml files
Date Wed, 26 Nov 2008 22:28:44 GMT

    [ https://issues.apache.org/jira/browse/SOLR-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651164#action_12651164
] 

Ruben Jimenez commented on SOLR-857:
------------------------------------

Ok I do take that back.  When I ran the same exact test as before with the exception of the
lucene core update, I still get a failure.  I ran the test against the entire set of files
this time with a larger max heap size and with fewer fields being stored.  At this point I'm
assuming that the schema changes are allowing the indexing process to proceed further than
it did before.  I'll run some tests with larger sets of files to get a sense of what the exact
limitation is when using a larger heap size.  

> Memory Leak during the indexing of large xml files
> --------------------------------------------------
>
>                 Key: SOLR-857
>                 URL: https://issues.apache.org/jira/browse/SOLR-857
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 1.3
>         Environment: Verified on Ubuntu 8.0.4 (1.7GB RAM, 2.4GHz dual core) and Windows
XP (2GB RAM, 2GHz pentium) both with a Java5 SDK
>            Reporter: Ruben Jimenez
>         Attachments: Dup_files1.zip, Dup_files10.zip, Dup_files2.zip, Dup_files3.zip,
Dup_files4.zip, Dup_files5.zip, Dup_files6.zip, Dup_files7.zip, Dup_files8.zip, Dup_files9.zip,
OQ_SOLR_00001.xml.zip, schema.xml, schema.xml.dup, solr.zip, solr256MBHeap.jpg
>
>
> While indexing a set of SOLR xml files that contain 5000 document adds within them and
are about 30MB each, SOLR 1.3 seems to continually use more and more memory until the heap
is exhausted, while the same files are indexed without issue with SOLR 1.2.
> Steps used to reproduce.
> 1 - Download SOLR 1.3
> 2 - Modify example schema.xml to match fields required
> 3 - start example server with following command java -Xms512m -Xmx1024m -XX:MaxPermSize=128m
-jar start.jar
> 4 - Index files as follow java -Xmx128m -jar .../examples/exampledocs/post.jar *.xml
> Directory with xml files contains about 100 xml files each of about 30MB each.  While
indexing after about the 25th file SOLR 1.3 runs out of memory, while SOLR 1.2 is able to
index the entire set of files without any problems.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message