incubator-connectors-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Connectors Framework > How to Build and Deploy ManifoldCF
Date Thu, 11 Nov 2010 17:06:00 GMT
Space: Apache Connectors Framework (https://cwiki.apache.org/confluence/display/CONNECTORS)
Page: How to Build and Deploy ManifoldCF (https://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+ManifoldCF)
Comment: https://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+ManifoldCF?focusedCommentId=24186050#comment-24186050

Comment added by Karl Wright:
---------------------------------------------------------------------

It's been a while since this was done with the ManifoldCF code base, but the MetaCarta code
base on which it is built regularly crawls 5 million or more without any issues of this kind. 
So the possibilities are:

(1) Misconfiguration of PostgreSQL

(2) Something about PostgreSQL 9.x.

(3) Something broken in the ManifoldCF code that got broken sometime in the last few months.\\

The obvious cross-check is for someone else (probably me) to do a large crawl while using
a properly configured PostgreSQL 8.x, and see where that leads.  Unfortunately I
probably won't have time to do that until next week, unless I can squeeze in a few moments
tonight.\\ \\

In reply to a comment by Farzad:
Using the default Null Output and File System connectors for this test.  I'm running on an
8 processor system, 8 GB of RAM, with 10,000 RPM drives.

So I seem to be in a pickle, cause I have 30 worker threads, and 400 allowed db connections,
I'm getting this error.  I'm going to set the db back down to 100 and see what happens.

I uploaded my configs to http://www.farzad.net/manifoldcf in case I over looked something.

Oh, the other thing, this happens around doc count of 60,000.  Have you tested with a very
large test set, perhaps 250,000 or 500,000?

Change your notification preferences: https://cwiki.apache.org/confluence/users/viewnotifications.action

Mime
View raw message