manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: max_pred_locks_per_transaction
Date Thu, 07 Feb 2013 18:57:43 GMT
I think I found the issue.

Please sync up and try again.
Karl


On Thu, Feb 7, 2013 at 9:43 AM, Karl Wright <daddywri@gmail.com> wrote:
> It is not necessary to wait until the job ends; each time it runs out
> of handles it prints the entire set of allocated handles.  You can
> stop the job now.
>
> Karl
>
> On Thu, Feb 7, 2013 at 9:41 AM, Erlend Garåsen <e.f.garasen@usit.uio.no> wrote:
>>
>> I will upload the log to Jira or place it on web as soon as the job ends.
>>
>> Erlend
>>
>>
>> On 07.02.13 15.34, Karl Wright wrote:
>>>
>>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>>> the handles that are currently considered to be in use.  Some of them
>>> are leaked, and some are just being used.  So it's really the sum
>>> total of all of them that will help us narrow down where leaks are
>>> occurring.
>>>
>>> I looked at the one location you provided; it's entirely unremarkable;
>>> it's not even happening in a transaction.  If similar locations show
>>> up again and again, it will mean that the problem must be in the part
>>> of the code that allocates connections at the lowest level.  Before I
>>> start picking that code apart line by line, though, I'd like all the
>>> data.
>>>
>>> Karl
>>>
>>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>>
>>>> Thanks, this is great.
>>>>
>>>> Could you include ALL of the 'possibly leaked' connections?  I'm
>>>> looking for the ones that show up over and over again.  Please attach
>>>> that part of the log to the ticket.
>>>>
>>>> Karl
>>>>
>>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e.f.garasen@usit.uio.no>
>>>> wrote:
>>>>>
>>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>>
>>>>>> FWIW, schema creation has been working fine here, and all tests that
>>>>>> exercise the schema have been passing fine as well.
>>>>>
>>>>>
>>>>>
>>>>> My fault, but everything is successfully set up now.
>>>>>
>>>>> I think I have reproduced the issue. The crawl is still running, but
I
>>>>> guess
>>>>> it will stop very soon due to the following entries in my log. The last
>>>>> fetched URL before the first of these log entries below was:
>>>>>
>>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>>
>>>>> It seems that this URL is part of a search, so maybe we will get around
>>>>> this
>>>>> problem by filtering out all search-related URLs. The problem is that
it
>>>>> is
>>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>>> that
>>>>> at next attempt.
>>>>>
>>>>>   INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>>
>>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>>>>   WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>>> connections,
>>>>> list of outstanding ones follows.
>>>>>   WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>>> leaked
>>>>> db connection
>>>>> java.lang.Exception: Possibly leaked db connection
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>>>
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>>>
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>>>>          at
>>>>>
>>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>>>>   WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>>> leaked
>>>>> db connection
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>>> 31050
>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message