manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erlend Garåsen <e.f.gara...@usit.uio.no>
Subject Re: max_pred_locks_per_transaction
Date Thu, 07 Feb 2013 20:14:48 GMT

Sorry for not following up with a thread dump shortly after, but I will 
if we still need one. I'm looking forward to try out the latest version 
from trunk tomorrow morning. It will be the first thing I'll do. Thanks 
for looking at this issue!

Erlend

On 07.02.13 19.57, Karl Wright wrote:
> I think I found the issue.
>
> Please sync up and try again.
> Karl
>
>
> On Thu, Feb 7, 2013 at 9:43 AM, Karl Wright <daddywri@gmail.com> wrote:
>> It is not necessary to wait until the job ends; each time it runs out
>> of handles it prints the entire set of allocated handles.  You can
>> stop the job now.
>>
>> Karl
>>
>> On Thu, Feb 7, 2013 at 9:41 AM, Erlend Garåsen <e.f.garasen@usit.uio.no> wrote:
>>>
>>> I will upload the log to Jira or place it on web as soon as the job ends.
>>>
>>> Erlend
>>>
>>>
>>> On 07.02.13 15.34, Karl Wright wrote:
>>>>
>>>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>>>> the handles that are currently considered to be in use.  Some of them
>>>> are leaked, and some are just being used.  So it's really the sum
>>>> total of all of them that will help us narrow down where leaks are
>>>> occurring.
>>>>
>>>> I looked at the one location you provided; it's entirely unremarkable;
>>>> it's not even happening in a transaction.  If similar locations show
>>>> up again and again, it will mean that the problem must be in the part
>>>> of the code that allocates connections at the lowest level.  Before I
>>>> start picking that code apart line by line, though, I'd like all the
>>>> data.
>>>>
>>>> Karl
>>>>
>>>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>>>
>>>>> Thanks, this is great.
>>>>>
>>>>> Could you include ALL of the 'possibly leaked' connections?  I'm
>>>>> looking for the ones that show up over and over again.  Please attach
>>>>> that part of the log to the ticket.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Garåsen <e.f.garasen@usit.uio.no>
>>>>> wrote:
>>>>>>
>>>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>>>
>>>>>>> FWIW, schema creation has been working fine here, and all tests
that
>>>>>>> exercise the schema have been passing fine as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>> My fault, but everything is successfully set up now.
>>>>>>
>>>>>> I think I have reproduced the issue. The crawl is still running,
but I
>>>>>> guess
>>>>>> it will stop very soon due to the following entries in my log. The
last
>>>>>> fetched URL before the first of these log entries below was:
>>>>>>
>>>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>>>
>>>>>> It seems that this URL is part of a search, so maybe we will get
around
>>>>>> this
>>>>>> problem by filtering out all search-related URLs. The problem is
that it
>>>>>> is
>>>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>>>> that
>>>>>> at next attempt.
>>>>>>
>>>>>>    INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>>>
>>>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=author_filter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|1360243927960+204|200|24992|
>>>>>>    WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>>>> connections,
>>>>>> list of outstanding ones follows.
>>>>>>    WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>>>> leaked
>>>>>> db connection
>>>>>> java.lang.Exception: Possibly leaked db connection
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:72)
>>>>>>
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:758)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
>>>>>>
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:674)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.java:649)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager.java:3777)
>>>>>>           at
>>>>>>
>>>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:224)
>>>>>>    WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>>>> leaked
>>>>>> db connection
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
VIP:
>>>>>> 31050
>>>
>>>
>>>
>>> --
>>> Erlend Garåsen
>>> Center for Information Technology Services
>>> University of Oslo
>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message