manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (CONNECTORS-224) OpenSearchServer connector
Date Wed, 10 Aug 2011 01:03:27 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082034#comment-13082034
] 

Karl Wright edited comment on CONNECTORS-224 at 8/10/11 1:02 AM:
-----------------------------------------------------------------

When I try to crawl using the connector and the connection parameters are bogus, the crawl
proceeds a while and does not seem to notice that there is no actual output server present.
 Very shortly thereafter, after a few hundred files, the whole system runs out of memory:

>>>>>>
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
        at java.lang.StringCoding.decode(StringCoding.java:173)
        at java.lang.StringCoding.decode(StringCoding.java:185)
        at java.lang.String.<init>(String.java:570)
        at java.lang.String.<init>(String.java:593)
        at org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServer
Index.getXML(OpenSearchServerIndex.java:58)
        at org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServer
Index.<init>(OpenSearchServerIndex.java:28)
        at org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServer
Connector.addOrReplaceDocument(OpenSearchServerConnector.java:153)
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.ad
dOrReplaceDocument(IncrementalIngester.java:1499)
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.pe
rformIngestion(IncrementalIngester.java:469)
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.do
cumentIngest(IncrementalIngester.java:364)
        at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ing
estDocument(WorkerThread.java:1587)
        at org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.pro
cessDocuments(FileConnector.java:283)
        at org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.proc
essDocuments(BaseRepositoryConnector.java:423)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.ja
va:564)
<<<<<<

I haven't looked at the code, but clearly the connector is trying to queue up documents in
memory for some reason.  If that's what's going on, that's never going to fly because there
cannot possibly be proper feedback of error conditions etc.

      was (Author: kwright@metacarta.com):
    When I try to crawl using the server and the connection is bogus, the crawl proceeds a
while and does not seem to notice that there is no output server.  Very shortly thereafter,
the whole system runs out of memory:

>>>>>>
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
        at java.lang.StringCoding.decode(StringCoding.java:173)
        at java.lang.StringCoding.decode(StringCoding.java:185)
        at java.lang.String.<init>(String.java:570)
        at java.lang.String.<init>(String.java:593)
        at org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServer
Index.getXML(OpenSearchServerIndex.java:58)
        at org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServer
Index.<init>(OpenSearchServerIndex.java:28)
        at org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServer
Connector.addOrReplaceDocument(OpenSearchServerConnector.java:153)
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.ad
dOrReplaceDocument(IncrementalIngester.java:1499)
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.pe
rformIngestion(IncrementalIngester.java:469)
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.do
cumentIngest(IncrementalIngester.java:364)
        at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ing
estDocument(WorkerThread.java:1587)
        at org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.pro
cessDocuments(FileConnector.java:283)
        at org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.proc
essDocuments(BaseRepositoryConnector.java:423)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.ja
va:564)
<<<<<<

I haven't looked at the code, but clearly the connector is trying to queue up documents in
memory for some reason.  If that's what's going on, that's never going to fly because there
cannot possibly be proper feedback of error conditions etc.
  
> OpenSearchServer connector
> --------------------------
>
>                 Key: CONNECTORS-224
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-224
>             Project: ManifoldCF
>          Issue Type: New Feature
>          Components: OpenSearchServer connector
>    Affects Versions: ManifoldCF 0.3
>            Reporter: Emmanuel Keller
>            Assignee: Karl Wright
>              Labels: OpenSearchServer, connector, outputconnector
>         Attachments: oss-mfc-alpha.patch, oss-mfc-alpha2.patch, oss-mfc-dev.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Provide an output connector for [OpenSearchServer|http://www.open-search-server.com].

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message