manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: max_pred_locks_per_transaction
Date Thu, 07 Feb 2013 14:43:27 GMT
It is not necessary to wait until the job ends; each time it runs out
of handles it prints the entire set of allocated handles.  You can
stop the job now.

Karl

On Thu, Feb 7, 2013 at 9:41 AM, Erlend Garåsen <e.f.garasen@usit.uio.no> wrote:
>
> I will upload the log to Jira or place it on web as soon as the job ends.
>
> Erlend
>
>
> On 07.02.13 15.34, Karl Wright wrote:
>>
>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>> the handles that are currently considered to be in use.  Some of them
>> are leaked, and some are just being used.  So it's really the sum
>> total of all of them that will help us narrow down where leaks are
>> occurring.
>>
>> I looked at the one location you provided; it's entirely unremarkable;
>> it's not even happening in a transaction.  If similar locations show
>> up again and again, it will mean that the problem must be in the part
>> of the code that allocates connections at the lowest level.  Before I
>> start picking that code apart line by line, though, I'd like all the
>> data.
>>
>> Karl
>>
>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>> Thanks, this is great.
>>>
>>> Could you include ALL of the 'possibly leaked' connections?  I'm
>>> looking for the ones that show up over and over again.  Please attach
>>> that part of the log to the ticket.
>>>
>>> Karl
>>>
>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e.f.garasen@usit.uio.no>
>>> wrote:
>>>>
>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>
>>>>> FWIW, schema creation has been working fine here, and all tests that
>>>>> exercise the schema have been passing fine as well.
>>>>
>>>>
>>>>
>>>> My fault, but everything is successfully set up now.
>>>>
>>>> I think I have reproduced the issue. The crawl is still running, but I
>>>> guess
>>>> it will stop very soon due to the following entries in my log. The last
>>>> fetched URL before the first of these log entries below was:
>>>>
>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>
>>>> It seems that this URL is part of a search, so maybe we will get around
>>>> this
>>>> problem by filtering out all search-related URLs. The problem is that it
>>>> is
>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>> that
>>>> at next attempt.
>>>>
>>>>   INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>
>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>>>   WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>> connections,
>>>> list of outstanding ones follows.
>>>>   WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>> leaked
>>>> db connection
>>>> java.lang.Exception: Possibly leaked db connection
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>>
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>>
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>>>   WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>> leaked
>>>> db connection
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message