manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: "Connection already in use" when crawling Windows Share repositories
Date Wed, 23 Feb 2011 14:39:37 GMT
I should also point out that setting up and tearing down connections
at high rates is what leads to a problem of lots of sockets in
TIME_WAIT.  The most elegant solution is to use a connection pool
rather than create and tear down connections on demand.  That way,
sockets are only really closed infrequently.

Karl


On Tue, Feb 22, 2011 at 6:09 AM, Karl Wright <daddywri@gmail.com> wrote:
> I've seen this before, when I was using HTTP and setting a connection
> up and tearing it down for each communication.  This is why HTTP
> keep-alive was invented in fact.
>
> TIME_WAIT is the result of an oddity in the way sockets are managed.
> A socket that has been closed but has not received an ACK from the
> other end is in this state.  Read here:
> http://stackoverflow.com/questions/813790/too-many-time-wait-connections
>
> If you can figure out what socket is being left dangling, then the
> right thing to do is create the socket using NO_WAIT, provided you
> have access to that code.  It's one of the socket parameters that you
> can use only at socket create time.  It tells the socket to not stick
> around after close.
>
> If this is in the ManifoldCF code, or in JCifs, I'd love to know about it! ;-)
>
> Thanks,
> Karl
>
>
> On Tue, Feb 22, 2011 at 4:40 AM,  <c.a.r.e@gmx.de> wrote:
>> Hi Karl,
>>
>> your guess was right. There has been a problem with too many connections in status
"TIME_WAIT".
>> Even if I set max. connections per JVM to 1 for the Repository and Authority Connection
I hat up to 3500 connections waiting.
>> When investingating this further I found out that an unbind response from the LDAP
server was missing.
>>
>> Sometimes this error occurs and sometimes it works fine. It's a bit weird, I wasn't
able to figure out where it came from.
>>
>> Thanks for your hint.
>>
>> Carina
>>
>>
>> -------- Original-Nachricht --------
>>> Datum: Fri, 18 Feb 2011 06:50:36 -0500
>>> Von: Karl Wright <daddywri@gmail.com>
>>> An: connectors-user@incubator.apache.org
>>> Betreff: Re: "Connection already in use" when crawling Windows Share repositories
>>
>>> I don't think this is directly a ManifoldCF issue.  My guess is that
>>> you are running out of file handles.  If you are on Windows, you might
>>> want to track the number of handles you are using via netstat.  If on
>>> linux, then lsof is the right tool.
>>>
>>> If you decide that this is indeed what the issue is, then on linux you
>>> need to either rebuild the kernel, or do stuff with ulimit to set the
>>> number of handles.  On Windows, I don't know how to do it.  Limiting
>>> the number of handles used by ManifoldCF is done by setting the number
>>> of connections under the "Throttling" tab of the various connection
>>> definitions you create, but that's not going to help with Tomcat.
>>>
>>> It may be worth checking if you aren't closing something.  That could
>>> easily cause this problem, eventually.  Since it is happening at
>>> crawling time, the Active Directory Authority is not actually
>>> communicating with the LDAP server yet.  JCIFS probably is, though.
>>>
>>> Let us know what you find.
>>>
>>> Thanks,
>>> Karl
>>>
>>>
>>> On Fri, Feb 18, 2011 at 6:31 AM,  <c.a.r.e@gmx.de> wrote:
>>> > I'm running MCF rev. 1071535 with Apache Tomcat 7. Solr and MCF are
>>> using HTTPS. When crawling data from a Windows Share Connection with Active
>>> Directory Authority after some time of crawling my job aborts with the status
>>> "Error: Couldn't set up SSL connection to ingestion API: Address already in
>>> use: connect".
>>> >
>>> > In the mcf logfile I get:
>>> >
>>> > [2011-02-18 11:24:33,695]ERROR Exception tossed: Couldn't set up SSL
>>> connection to ingestion API: Address already in use: connect
>>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: Couldn't set
>>> up SSL connection to ingestion API: Address already in use: connect
>>> >        at
>>> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:736)
>>> >        at
>>> org.apache.manifoldcf.agents.output.solr.HttpPoster$IngestThread.run(HttpPoster.java:1006)
>>> > Caused by: java.net.BindException: Address already in use: connect
>>> >        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> >        at
>>> java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>> >        at
>>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>> >        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>> >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>> >        at java.net.Socket.connect(Socket.java:529)
>>> >        at
>>> com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:550)
>>> >        at
>>> com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:353)
>>> >        at
>>> com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:71)
>>> >        at
>>> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:728)
>>> >        ... 1 more
>>> >
>>> > I use the same LDAP user
>>> > - to authenticate at MCF
>>> > - login credentials for the Solr Connection
>>> > - credentials for authority connection
>>> > - and for connecting to our LDAP server itself (in tomcat JNDI Realm
>>> config).
>>> >
>>> > It seems to me that the connection to LDAP might be a problem. I tried
>>> to configure connection pooling for tomcat. (I'm afraid I'm not so familiar
>>> with that.)
>>> > "-Dcom.sun.jndi.ldap.connect.pool=true
>>> -Dcom.jndi.ldap.connect.pool.protocol=ssl"
>>> >
>>> > Did I miss sth in the config? Or how do I have to configure MCF?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Carina
>>> > --
>>> > GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
>>> > gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
>>> >
>>
>> --
>> GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
>> gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
>>
>

Mime
View raw message