incubator-connectors-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Connectors Framework > How to Build and Deploy ManifoldCF
Date Thu, 11 Nov 2010 16:35:00 GMT
Space: Apache Connectors Framework (https://cwiki.apache.org/confluence/display/CONNECTORS)
Page: How to Build and Deploy ManifoldCF (https://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+ManifoldCF)
Comment: https://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+ManifoldCF?focusedCommentId=24186046#comment-24186046

Comment added by Farzad:
---------------------------------------------------------------------

Using the default Null Output and File System connectors for this test.  I'm running on an
8 processor system, 8 GB of RAM, with 10,000 RPM drives.

So I seem to be in a pickle, cause I have 30 worker threads, and 400 allowed db connections,
I'm getting this error.  I'm going to set the db back down to 100 and see what happens.

I uploaded my configs to http://www.farzad.net/manifoldcf in case I over looked something.

Oh, the other thing, this happens around doc count of 60,000.  Have you tested with a very
large test set, perhaps 250,000 or 500,000?

In reply to a comment by Karl Wright:
You do not need to increase the number of worker threads, unless you have a machine capable
of massive parallelism, in which case I could have sworn the instructions said it was OK to
increase the number of worker threads to around 100.  If you do that, you should have a postgresql
max connections parameter of 300 or 400.

The instructions for the postgresql parameters are specific to some extent to postgresql 8.x.
 I don't know what they should be on 9.x.  I can only tell you that your symptom is occurring
because ManifoldCF is trying to grab more connections that PostgreSQL is willing to give.
 Since ManifoldCF uses a connection pool, it can only grab one connection per thread.

The only way this would be violated is if a connector was written that uses connection-specific
database tables and doesn't properly return handles to the pool after grabbing them.  Is there
a possibility that this is occurring?

Change your notification preferences: https://cwiki.apache.org/confluence/users/viewnotifications.action

Mime
View raw message