Mailing-List: contact solr-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-dev@lucene.apache.org
Message-ID: <314343149.1259963240672.JavaMail.jira@brutus>
Date: Fri, 4 Dec 2009 21:47:20 +0000 (UTC)
From: "Laurent Chavet (JIRA)" <jira@apache.org>
To: solr-dev@lucene.apache.org
Subject: [jira] Commented: (SOLR-1623) Solr hangs (often throwing
 java.lang.OutOfMemoryError: PermGen space) when indexing many different
 field names
In-Reply-To: <1454624508.1259959400812.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SOLR-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786185#action_12786185 ] 

Laurent Chavet commented on SOLR-1623:
--------------------------------------

Yes this definitely repros in 1.4.

Unfortunately I think I need a lot of fields; here is what I am trying to do:

I want to store news articles and extract many topics for each story with a score for each topic for each story.

So for example a story migh have a topic of Crime with a score of 20.

So what I am doing now is store:

Field:Topic    Value:Crime      indexed="true" stored="true"            (need to searched and retrieved)
Field:Weight_Topic_Crime  Value:20  indexed="true" stored="true"   (needs to be sorted and retrieved)

Because there can be a lot of different value for the field topic; with this schema  we end up with a lot of fields starting with weight.

Any suggestion on how to achieve the same result in a different way?

Thanks,

Laurent

> Solr hangs (often throwing java.lang.OutOfMemoryError: PermGen space) when indexing many different field names
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-1623
>                 URL: https://issues.apache.org/jira/browse/SOLR-1623
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.3, 1.4
>         Environment: Tomcat Version         JVM Version                      JVM Vendor                    OS Name OS Version        OS Architecture 
> Apache Tomcat/6.0   snapshot 1.6.0_13-b03 Sun Microsystems Inc. Linux         2.6.18-164.el5  amd64 
> and/or
> Tomcat Version                JVM Version         JVM Vendor                        OS Name               OS Version    OS Architecture 
> Apache Tomcat/6.0.18   1.6.0_12-b04        Sun Microsystems Inc.     Windows 2003     5.2                   amd64 
>            Reporter: Laurent Chavet
>            Priority: Critical
>
> With the following fields in schema.xml:
>  <fields>
>    <field name="id" type="sint" indexed="true" stored="true" required="true" /> 
>     <dynamicField name="weight_*"  type="sint"    indexed="true"  stored="true"/>
> </fields>
> Run the following code:
> import java.util.ArrayList;
> import java.util.List;
> import org.apache.solr.client.solrj.SolrServer;
> import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer;
> import org.apache.solr.common.SolrInputDocument;
>     public static void main(String[] args) throws Exception {
>         SolrServer server;
>         try {
>             server = new CommonsHttpSolrServer(args[0]);
>         } catch (Exception e) {
>             System.err.println("can't creater server using: " + args[0] + "  " + e.getMessage());
>             throw e;
>         }
>         for (int i = 0; i < 1000; i++) {
>             List<SolrInputDocument> batchedDocs = new ArrayList<SolrInputDocument>();
>             for (int j = 0; j < 1000; j++) {
>                 SolrInputDocument doc = new SolrInputDocument();
>                 doc.addField("id", i * 1000 + j);
>                 // hangs after 30 to 50 batches
>                 doc.addField("weight_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + Integer.toString(i) + "_" + Integer.toString(j), i * 1000 + j);
>                 // hangs after about 200 batches
>                 //doc.addField("weight_" + Integer.toString(i) + "_" + Integer.toString(j), i * 1000 + j);
>                 batchedDocs.add(doc);
>             }
>             try {
>                 server.add(batchedDocs, true);
>                 System.err.println("Done with batch=" + i);
>                 // server.commit(); //doesn't change anything
>             } catch (Exception e) {
>                 System.err.println("batchId=" + i + " bad batch: " + e.getMessage());
>                 throw e;
>             }
>         }
>     }
> And soon the client (sometime throws) and solr will freeze. sometime you can see: java.lang.OutOfMemoryError: PermGen space in the server logs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.