Mailing-List: contact user-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@manifoldcf.apache.org
Received-SPF: pass (athena.apache.org: domain of daddywri@gmail.com designates
 209.85.128.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <5113BD34.6020901@usit.uio.no>
References: <51025DC7.8000702@usit.uio.no>
	<CALUFAGAU4Wc+QqQ2xy96jJSJx_ZfcC0MXqiKqysr-TsJFy1i4A@mail.gmail.com>
	<CALUFAGAfPrQQW-gJE5fwUncuWE4T9rfUGz8miS2oVxTfU8r4yA@mail.gmail.com>
	<5110FD4A.6040006@usit.uio.no>
	<CALUFAGDXhda2Aob8if=Q3mEAaqcXK09SYq9TD_5Dprsd+fGKQg@mail.gmail.com>
	<51122706.2040709@usit.uio.no>
	<CALUFAGBjNRK=uBMs_6y=+4MUw8Dc+=5zAeQ_UVRB4Q9C9fi_TA@mail.gmail.com>
	<511244E4.4020102@usit.uio.no>
	<51124947.7080309@usit.uio.no>
	<CALUFAGDtENwFxk613NSzSGV=dorwuXaEc064rNZxOpr2vKnogQ@mail.gmail.com>
	<511251AE.3050805@usit.uio.no>
	<CALUFAGC6+KFO-+rFt06oqdQ6FNGbvCHET0CRFDB62Sik6dbvDw@mail.gmail.com>
	<511255A6.4070908@usit.uio.no>
	<CALUFAGDSwcNmb9PXNVwe11fRGQLxVwfNVz+FZfYgKUZQv4YJFA@mail.gmail.com>
	<51126252.1030506@usit.uio.no>
	<CALUFAGCO+CoyJqD19F7wsB_zzvdFvKeZnNTaAvSPW6SpkEnd0A@mail.gmail.com>
	<5113B485.6070202@usit.uio.no>
	<CALUFAGAEMqi6U4+Agid_i2W1J3sQcYM-P0u-huRLTH=NCSX0pQ@mail.gmail.com>
	<CALUFAGAC5agf_REARuerv5Gpw+-WFRK_sT5iDCERczwSipt37A@mail.gmail.com>
	<5113BD34.6020901@usit.uio.no>
Date: Thu, 7 Feb 2013 09:43:27 -0500
Message-ID: 
 <CALUFAGAFsCVmq=6B77CO+xenY-sK8Gq8PrcfRN_DS_pTOeQV0w@mail.gmail.com>
Subject: Re: max_pred_locks_per_transaction
From: Karl Wright <daddywri@gmail.com>
To: user@manifoldcf.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

It is not necessary to wait until the job ends; each time it runs out
of handles it prints the entire set of allocated handles.  You can
stop the job now.

Karl

On Thu, Feb 7, 2013 at 9:41 AM, Erlend Gar=E5sen <e.f.garasen@usit.uio.no> =
wrote:
>
> I will upload the log to Jira or place it on web as soon as the job ends.
>
> Erlend
>
>
> On 07.02.13 15.34, Karl Wright wrote:
>>
>> FWIW, the way this diagnostic code works is that it simply dumps ALL
>> the handles that are currently considered to be in use.  Some of them
>> are leaked, and some are just being used.  So it's really the sum
>> total of all of them that will help us narrow down where leaks are
>> occurring.
>>
>> I looked at the one location you provided; it's entirely unremarkable;
>> it's not even happening in a transaction.  If similar locations show
>> up again and again, it will mean that the problem must be in the part
>> of the code that allocates connections at the lowest level.  Before I
>> start picking that code apart line by line, though, I'd like all the
>> data.
>>
>> Karl
>>
>> On Thu, Feb 7, 2013 at 9:07 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>> Thanks, this is great.
>>>
>>> Could you include ALL of the 'possibly leaked' connections?  I'm
>>> looking for the ones that show up over and over again.  Please attach
>>> that part of the log to the ticket.
>>>
>>> Karl
>>>
>>> On Thu, Feb 7, 2013 at 9:04 AM, Erlend Gar=E5sen <e.f.garasen@usit.uio.=
no>
>>> wrote:
>>>>
>>>> On 06.02.13 15.24, Karl Wright wrote:
>>>>
>>>>> FWIW, schema creation has been working fine here, and all tests that
>>>>> exercise the schema have been passing fine as well.
>>>>
>>>>
>>>>
>>>> My fault, but everything is successfully set up now.
>>>>
>>>> I think I have reproduced the issue. The crawl is still running, but I
>>>> guess
>>>> it will stop very soon due to the following entries in my log. The las=
t
>>>> fetched URL before the first of these log entries below was:
>>>>
>>>> https://www.duo.uio.no/handle/123456789/31/discover?fq=3Dauthor_filter=
%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid
>>>>
>>>> It seems that this URL is part of a search, so maybe we will get aroun=
d
>>>> this
>>>> problem by filtering out all search-related URLs. The problem is that =
it
>>>> is
>>>> already filtered out, but by doing post filtering. I can try to adjust
>>>> that
>>>> at next attempt.
>>>>
>>>>   INFO 2013-02-07 14:32:08,166 (Worker thread '13') - WEB: FETCH
>>>>
>>>> URL|https://www.duo.uio.no/handle/123456789/31/discover?fq=3Dauthor_fi=
lter%3Aaaberge%2C%5C+astrid%5C%7C%5C%7C%5C%7CAaberge%2C%5C+Astrid|136024392=
7960+204|200|24992|
>>>>   WARN 2013-02-07 14:32:08,228 (Worker thread '16') - Out of db
>>>> connections,
>>>> list of outstanding ones follows.
>>>>   WARN 2013-02-07 14:32:08,229 (Worker thread '16') - Found a possibly
>>>> leaked
>>>> db connection
>>>> java.lang.Exception: Possibly leaked db connection
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(Conne=
ctionPool.java:72)
>>>>
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWit=
hRetries(ConnectionFactory.java:125)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(Co=
nnectionFactory.java:96)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Data=
base.java:758)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create=
(Database.java:1430)
>>>>
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExe=
cute(CacheManager.java:144)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.jav=
a:186)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery=
(DBInterfacePostgreSQL.java:803)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.j=
ava:221)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.HopCount.processFind(HopCount.java:=
674)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.HopCount.findHopCounts(HopCount.jav=
a:649)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.findHopCounts(JobManager=
.java:3777)
>>>>          at
>>>>
>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.jav=
a:224)
>>>>   WARN 2013-02-07 14:32:08,233 (Worker thread '16') - Found a possibly
>>>> leaked
>>>> db connection
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Gar=E5sen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>
>
>
> --
> Erlend Gar=E5sen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 310=
50