manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-1562) Documents unreachable due to hopcount are not considered unreachable on cleanup pass
Date Fri, 14 Dec 2018 06:22:00 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720982#comment-16720982
] 

Karl Wright commented on CONNECTORS-1562:
-----------------------------------------

Attached the "reduced" step with query logging.  Analysis will take some time.  The entire
startup log chunk is here (and it contains the seeding part, which is what we're interested
in):

{code}
DEBUG 2018-12-14T01:07:42,367 (Startup thread) - Requested query: [UPDATE jobqueue SET needpriorityprocessid=NULL,needpriority=?
WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,368 (Thread-688) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,368 (Thread-688) - Done actual query (0ms): [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Actual query: [UPDATE jobqueue SET needpriorityprocessid=NULL,needpriority=?
WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,369 (Thread-689) -   Parameter 0: 'T'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) -   Parameter 2: 'P'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) -   Parameter 3: '1544121003866'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) -   Parameter 4: 'G'
DEBUG 2018-12-14T01:07:42,369 (Thread-689) - Done actual query (0ms): [UPDATE jobqueue SET
needpriorityprocessid=NULL,needpriority=? WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,369 (Startup thread) - Beginning transaction of type 2
DEBUG 2018-12-14T01:07:42,369 (Startup thread) - Marking for delete for job 1544121003866
all hopcount document references from table jobqueue t99 matching t99.status IN (?,?)
DEBUG 2018-12-14T01:07:42,370 (Startup thread) - Requested query: [UPDATE hopcount SET distance=?,deathmark=?
WHERE id IN(SELECT t0.ownerid FROM hopdeletedeps t0,jobqueue t99,intrinsiclink t1 WHERE t0.jobid=?
AND (t1.jobid=? AND t1.parentidhash=t0.parentidhash AND t1.linktype=t0.linktype AND t1.childidhash=t0.childidhash)
AND (t99.jobid=? AND t99.dochash=t0.childidhash) AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,370 (Thread-690) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,370 (Thread-690) - Done actual query (0ms): [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,371 (Thread-691) - Actual query: [UPDATE hopcount SET distance=?,deathmark=?
WHERE id IN(SELECT t0.ownerid FROM hopdeletedeps t0,jobqueue t99,intrinsiclink t1 WHERE t0.jobid=?
AND (t1.jobid=? AND t1.parentidhash=t0.parentidhash AND t1.linktype=t0.linktype AND t1.childidhash=t0.childidhash)
AND (t99.jobid=? AND t99.dochash=t0.childidhash) AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 0: '-1'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 3: '1544121003866'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 4: '1544121003866'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 5: 'P'
DEBUG 2018-12-14T01:07:42,371 (Thread-691) -   Parameter 6: 'H'
DEBUG 2018-12-14T01:07:42,389 (Thread-691) - Done actual query (18ms): [UPDATE hopcount SET
distance=?,deathmark=? WHERE id IN(SELECT t0.ownerid FROM hopdeletedeps t0,jobqueue t99,intrinsiclink
t1 WHERE t0.jobid=? AND (t1.jobid=? AND t1.parentidhash=t0.parentidhash AND t1.linktype=t0.linktype
AND t1.childidhash=t0.childidhash) AND (t99.jobid=? AND t99.dochash=t0.childidhash) AND t99.status
IN (?,?))]
DEBUG 2018-12-14T01:07:42,390 (Startup thread) - Done setting hopcount rows for job 1544121003866
to initial distances
DEBUG 2018-12-14T01:07:42,390 (Startup thread) - Requested query: [DELETE FROM intrinsiclink
WHERE EXISTS(SELECT 'x' FROM jobqueue t99 WHERE (t99.jobid=? AND t99.dochash=intrinsiclink.childidhash)
AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,390 (Thread-692) - Actual query: [DELETE FROM intrinsiclink WHERE
EXISTS(SELECT 'x' FROM jobqueue t99 WHERE (t99.jobid=? AND t99.dochash=intrinsiclink.childidhash)
AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,391 (Thread-692) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,391 (Thread-692) -   Parameter 1: 'P'
DEBUG 2018-12-14T01:07:42,391 (Thread-692) -   Parameter 2: 'H'
DEBUG 2018-12-14T01:07:42,407 (Thread-692) - Done actual query (17ms): [DELETE FROM intrinsiclink
WHERE EXISTS(SELECT 'x' FROM jobqueue t99 WHERE (t99.jobid=? AND t99.dochash=intrinsiclink.childidhash)
AND t99.status IN (?,?))]
DEBUG 2018-12-14T01:07:42,408 (Startup thread) - Requested query: [DELETE FROM hopdeletedeps
WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND deathmark=?))]
DEBUG 2018-12-14T01:07:42,408 (Thread-693) - Actual query: [DELETE FROM hopdeletedeps WHERE
ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND deathmark=?))]
DEBUG 2018-12-14T01:07:42,409 (Thread-693) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,409 (Thread-693) -   Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,410 (Thread-693) - Done actual query (2ms): [DELETE FROM hopdeletedeps
WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND deathmark=?))]
DEBUG 2018-12-14T01:07:42,410 (Startup thread) - Requested query: [UPDATE hopcount SET deathmark=?
WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Actual query: [UPDATE hopcount SET deathmark=?
WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,411 (Thread-694) -   Parameter 0: 'Q'
DEBUG 2018-12-14T01:07:42,411 (Thread-694) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,411 (Thread-694) -   Parameter 2: 'D'
DEBUG 2018-12-14T01:07:42,411 (Thread-694) - Done actual query (0ms): [UPDATE hopcount SET
deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,411 (Startup thread) - Done queueing for deletion for 1544121003866
DEBUG 2018-12-14T01:07:42,411 (Startup thread) - Requested query: [DELETE FROM prereqevents
WHERE EXISTS(SELECT 'x' FROM jobqueue t0 WHERE t0.id=prereqevents.owner AND (t0.jobid=? AND
t0.status=? OR t0.jobid=? AND t0.status=?))]
DEBUG 2018-12-14T01:07:42,412 (Thread-695) - Actual query: [DELETE FROM prereqevents WHERE
EXISTS(SELECT 'x' FROM jobqueue t0 WHERE t0.id=prereqevents.owner AND (t0.jobid=? AND t0.status=?
OR t0.jobid=? AND t0.status=?))]
DEBUG 2018-12-14T01:07:42,412 (Thread-695) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,412 (Thread-695) -   Parameter 1: 'P'
DEBUG 2018-12-14T01:07:42,412 (Thread-695) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,412 (Thread-695) -   Parameter 3: 'H'
DEBUG 2018-12-14T01:07:42,419 (Thread-695) - Done actual query (7ms): [DELETE FROM prereqevents
WHERE EXISTS(SELECT 'x' FROM jobqueue t0 WHERE t0.id=prereqevents.owner AND (t0.jobid=? AND
t0.status=? OR t0.jobid=? AND t0.status=?))]
DEBUG 2018-12-14T01:07:42,419 (Startup thread) - Requested query: [DELETE FROM jobqueue WHERE
(jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,420 (Thread-696) - Actual query: [DELETE FROM jobqueue WHERE (jobid=?
AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,420 (Thread-696) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,420 (Thread-696) -   Parameter 1: 'P'
DEBUG 2018-12-14T01:07:42,420 (Thread-696) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,420 (Thread-696) -   Parameter 3: 'H'
DEBUG 2018-12-14T01:07:42,421 (Thread-696) - Done actual query (2ms): [DELETE FROM jobqueue
WHERE (jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,421 (Startup thread) - Requested query: [UPDATE jobqueue SET failtime=NULL,checktime=?,failcount=NULL,checkaction=NULL,status=?
WHERE (jobid=? AND status=? OR jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,422 (Thread-697) - Actual query: [UPDATE jobqueue SET failtime=NULL,checktime=?,failcount=NULL,checkaction=NULL,status=?
WHERE (jobid=? AND status=? OR jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 0: '0'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 1: 'Z'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 3: 'G'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 4: '1544121003866'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 5: 'U'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 6: '1544121003866'
DEBUG 2018-12-14T01:07:42,422 (Thread-697) -   Parameter 7: 'C'
DEBUG 2018-12-14T01:07:42,423 (Thread-697) - Done actual query (1ms): [UPDATE jobqueue SET
failtime=NULL,checktime=?,failcount=NULL,checkaction=NULL,status=? WHERE (jobid=? AND status=?
OR jobid=? AND status=? OR jobid=? AND status=?)]
DEBUG 2018-12-14T01:07:42,423 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,439 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,454 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,454 (Startup thread) - Requested query: [SELECT bincounter FROM
docbins WHERE (connectorclass=? AND binname=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,454 (Thread-698) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,454 (Thread-698) - Done actual query (0ms): [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,455 (Thread-699) - Actual query: [SELECT bincounter FROM docbins
WHERE (connectorclass=? AND binname=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,455 (Thread-699) -   Parameter 0: 'org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector'
DEBUG 2018-12-14T01:07:42,455 (Thread-699) -   Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,455 (Thread-699) - Done actual query (0ms): [SELECT bincounter FROM
docbins WHERE (connectorclass=? AND binname=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,455 (Startup thread) - Requested query: [INSERT INTO docbins (connectorclass,bincounter,binname)
VALUES (?,?,?)]
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Actual query: [INSERT INTO docbins (connectorclass,bincounter,binname)
VALUES (?,?,?)]
DEBUG 2018-12-14T01:07:42,456 (Thread-700) -   Parameter 0: 'org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector'
DEBUG 2018-12-14T01:07:42,456 (Thread-700) -   Parameter 1: '5.0'
DEBUG 2018-12-14T01:07:42,456 (Thread-700) -   Parameter 2: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,456 (Thread-700) - Done actual query (0ms): [INSERT INTO docbins
(connectorclass,bincounter,binname) VALUES (?,?,?)]
DEBUG 2018-12-14T01:07:42,456 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,457 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,457 (Startup thread) - Beginning transaction of type 2
DEBUG 2018-12-14T01:07:42,457 (Startup thread) - Requested query: [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,458 (Thread-701) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,458 (Thread-701) - Done actual query (0ms): [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,458 (Thread-702) - Actual query: [SELECT id,status,checktime FROM
jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,458 (Thread-702) -   Parameter 0: '00EFFBADCA2DC5EA9A03967A85DBCD87090B05B2'
DEBUG 2018-12-14T01:07:42,458 (Thread-702) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,459 (Thread-702) - Done actual query (1ms): [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,460 (Startup thread) - Requested query: [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,460 (Thread-703) - Actual query: [DELETE FROM prereqevents  WHERE
owner=?]
DEBUG 2018-12-14T01:07:42,460 (Thread-703) -   Parameter 0: '1544767286064'
DEBUG 2018-12-14T01:07:42,461 (Thread-703) - Done actual query (1ms): [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,461 (Startup thread) - Requested query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,461 (Thread-704) - Actual query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 0: '0.0'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,461 (Thread-704) -   Parameter 7: '1544767286064'
DEBUG 2018-12-14T01:07:42,462 (Thread-704) - Done actual query (1ms): [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,463 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,463 (Thread-705) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,463 (Thread-705) -   Parameter 0: '1544767286064'
DEBUG 2018-12-14T01:07:42,463 (Thread-705) -   Parameter 1: ':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,463 (Thread-705) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,463 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,464 (Thread-706) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,464 (Thread-706) -   Parameter 0: '1544767286064'
DEBUG 2018-12-14T01:07:42,464 (Thread-706) -   Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,464 (Thread-706) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,465 (Startup thread) - Requested query: [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,465 (Thread-707) - Actual query: [SELECT id,status,checktime FROM
jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,465 (Thread-707) -   Parameter 0: '1B901235B7D91C61B5F50851437A10527E97ECF3'
DEBUG 2018-12-14T01:07:42,465 (Thread-707) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,466 (Thread-707) - Done actual query (1ms): [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,466 (Startup thread) - Requested query: [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,466 (Thread-708) - Actual query: [DELETE FROM prereqevents  WHERE
owner=?]
DEBUG 2018-12-14T01:07:42,466 (Thread-708) -   Parameter 0: '1544767286063'
DEBUG 2018-12-14T01:07:42,467 (Thread-708) - Done actual query (1ms): [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,467 (Startup thread) - Requested query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,467 (Thread-709) - Actual query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 0: '0.6931471805599453'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,467 (Thread-709) -   Parameter 7: '1544767286063'
DEBUG 2018-12-14T01:07:42,468 (Thread-709) - Done actual query (1ms): [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,468 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,468 (Thread-710) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,468 (Thread-710) -   Parameter 0: '1544767286063'
DEBUG 2018-12-14T01:07:42,468 (Thread-710) -   Parameter 1: ':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,468 (Thread-710) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Thread-711) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Thread-711) -   Parameter 0: '1544767286063'
DEBUG 2018-12-14T01:07:42,469 (Thread-711) -   Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,469 (Thread-711) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,469 (Startup thread) - Requested query: [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,469 (Thread-712) - Actual query: [SELECT id,status,checktime FROM
jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,469 (Thread-712) -   Parameter 0: '3B08425D39C140EB4403190B7481C7A90A8435FF'
DEBUG 2018-12-14T01:07:42,469 (Thread-712) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,470 (Thread-712) - Done actual query (1ms): [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,470 (Startup thread) - Requested query: [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-713) - Actual query: [DELETE FROM prereqevents  WHERE
owner=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-713) -   Parameter 0: '1544767286062'
DEBUG 2018-12-14T01:07:42,471 (Thread-713) - Done actual query (0ms): [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,471 (Startup thread) - Requested query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-714) - Actual query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,471 (Thread-714) -   Parameter 0: '1.0986122886681098'
DEBUG 2018-12-14T01:07:42,471 (Thread-714) -   Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) -   Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) -   Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) -   Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,472 (Thread-714) -   Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) -   Parameter 7: '1544767286062'
DEBUG 2018-12-14T01:07:42,472 (Thread-714) - Done actual query (1ms): [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,472 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,473 (Thread-715) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,473 (Thread-715) -   Parameter 0: '1544767286062'
DEBUG 2018-12-14T01:07:42,473 (Thread-715) -   Parameter 1: ':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,473 (Thread-715) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,473 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,474 (Thread-716) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,474 (Thread-716) -   Parameter 0: '1544767286062'
DEBUG 2018-12-14T01:07:42,474 (Thread-716) -   Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,474 (Thread-716) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,474 (Startup thread) - Requested query: [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,475 (Thread-717) - Actual query: [SELECT id,status,checktime FROM
jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,475 (Thread-717) -   Parameter 0: '9DC7778DE94DBCD1BD2A7D7BAEFAB09919972A78'
DEBUG 2018-12-14T01:07:42,475 (Thread-717) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,476 (Thread-717) - Done actual query (1ms): [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,476 (Startup thread) - Requested query: [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,476 (Thread-718) - Actual query: [DELETE FROM prereqevents  WHERE
owner=?]
DEBUG 2018-12-14T01:07:42,476 (Thread-718) -   Parameter 0: '1544767286059'
DEBUG 2018-12-14T01:07:42,477 (Thread-718) - Done actual query (1ms): [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,477 (Startup thread) - Requested query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,477 (Thread-719) - Actual query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 0: '1.3862943611198906'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,477 (Thread-719) -   Parameter 7: '1544767286059'
DEBUG 2018-12-14T01:07:42,478 (Thread-719) - Done actual query (1ms): [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,478 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,478 (Thread-720) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,478 (Thread-720) -   Parameter 0: '1544767286059'
DEBUG 2018-12-14T01:07:42,478 (Thread-720) -   Parameter 1: ':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,479 (Thread-720) - Done actual query (1ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,479 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,479 (Thread-721) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,479 (Thread-721) -   Parameter 0: '1544767286059'
DEBUG 2018-12-14T01:07:42,479 (Thread-721) -   Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,480 (Thread-721) - Done actual query (1ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,480 (Startup thread) - Requested query: [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,480 (Thread-722) - Actual query: [SELECT id,status,checktime FROM
jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,480 (Thread-722) -   Parameter 0: 'BCE78F106BCB6F912A5A10B55D182A53504AA6C2'
DEBUG 2018-12-14T01:07:42,480 (Thread-722) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,480 (Thread-722) - Done actual query (0ms): [SELECT id,status,checktime
FROM jobqueue WHERE (dochash=? AND jobid=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,481 (Startup thread) - Requested query: [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,481 (Thread-723) - Actual query: [DELETE FROM prereqevents  WHERE
owner=?]
DEBUG 2018-12-14T01:07:42,481 (Thread-723) -   Parameter 0: '1544767286058'
DEBUG 2018-12-14T01:07:42,481 (Thread-723) - Done actual query (0ms): [DELETE FROM prereqevents
 WHERE owner=?]
DEBUG 2018-12-14T01:07:42,481 (Startup thread) - Requested query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Actual query: [UPDATE jobqueue SET docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 0: '1.6094379124341003'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 1: '0'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 2: 'F'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 3: 'R'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 6: 'G'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) -   Parameter 7: '1544767286058'
DEBUG 2018-12-14T01:07:42,482 (Thread-724) - Done actual query (0ms): [UPDATE jobqueue SET
docpriority=?,failtime=NULL,checktime=?,failcount=NULL,needpriority=?,checkaction=?,isseed=?,seedingprocessid=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,482 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,483 (Thread-725) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,483 (Thread-725) -   Parameter 0: '1544767286058'
DEBUG 2018-12-14T01:07:42,483 (Thread-725) -   Parameter 1: ':webcrawler:robots:https:www.uantwerpen.be:443'
DEBUG 2018-12-14T01:07:42,483 (Thread-725) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,483 (Startup thread) - Requested query: [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,484 (Thread-726) - Actual query: [INSERT INTO prereqevents (owner,eventname)
VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,484 (Thread-726) -   Parameter 0: '1544767286058'
DEBUG 2018-12-14T01:07:42,484 (Thread-726) -   Parameter 1: 'www.uantwerpen.be'
DEBUG 2018-12-14T01:07:42,484 (Thread-726) - Done actual query (0ms): [INSERT INTO prereqevents
(owner,eventname) VALUES (?,?)]
DEBUG 2018-12-14T01:07:42,484 (Startup thread) - Requested query: [SELECT parentidhash FROM
intrinsiclink WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=?
AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=?
AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=?
AND parentidhash=? AND linktype=? AND childidhash=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,485 (Thread-727) - Actual query: [SELECT parentidhash FROM intrinsiclink
WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND parentidhash=?
AND linktype=? AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=?
OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND parentidhash=?
AND linktype=? AND childidhash=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 1: '00EFFBADCA2DC5EA9A03967A85DBCD87090B05B2'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 2: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 3: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 4: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 5: '1B901235B7D91C61B5F50851437A10527E97ECF3'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 6: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 7: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 8: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 9: '3B08425D39C140EB4403190B7481C7A90A8435FF'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 10: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 11: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 12: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 13: '9DC7778DE94DBCD1BD2A7D7BAEFAB09919972A78'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 14: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 15: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 16: '1544121003866'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 17: 'BCE78F106BCB6F912A5A10B55D182A53504AA6C2'
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 18: ''
DEBUG 2018-12-14T01:07:42,485 (Thread-727) -   Parameter 19: ''
DEBUG 2018-12-14T01:07:42,486 (Thread-727) - Done actual query (1ms): [SELECT parentidhash
FROM intrinsiclink WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=?
AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=?
AND childidhash=? OR jobid=? AND parentidhash=? AND linktype=? AND childidhash=? OR jobid=?
AND parentidhash=? AND linktype=? AND childidhash=?) FOR UPDATE]
DEBUG 2018-12-14T01:07:42,486 (Startup thread) - Requested query: [UPDATE intrinsiclink SET
processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Actual query: [UPDATE intrinsiclink SET processid=?,isnew=?
WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,487 (Thread-728) -   Parameter 0: ''
DEBUG 2018-12-14T01:07:42,487 (Thread-728) -   Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,487 (Thread-728) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,487 (Thread-728) -   Parameter 3: '3B08425D39C140EB4403190B7481C7A90A8435FF'
DEBUG 2018-12-14T01:07:42,487 (Thread-728) -   Parameter 4: ''
DEBUG 2018-12-14T01:07:42,487 (Thread-728) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,487 (Thread-728) - Done actual query (1ms): [UPDATE intrinsiclink
SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,487 (Startup thread) - Requested query: [UPDATE intrinsiclink SET
processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Actual query: [UPDATE intrinsiclink SET processid=?,isnew=?
WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,488 (Thread-729) -   Parameter 0: ''
DEBUG 2018-12-14T01:07:42,488 (Thread-729) -   Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,488 (Thread-729) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,488 (Thread-729) -   Parameter 3: '1B901235B7D91C61B5F50851437A10527E97ECF3'
DEBUG 2018-12-14T01:07:42,488 (Thread-729) -   Parameter 4: ''
DEBUG 2018-12-14T01:07:42,488 (Thread-729) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,488 (Thread-729) - Done actual query (0ms): [UPDATE intrinsiclink
SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,488 (Startup thread) - Requested query: [UPDATE intrinsiclink SET
processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Actual query: [UPDATE intrinsiclink SET processid=?,isnew=?
WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,489 (Thread-730) -   Parameter 0: ''
DEBUG 2018-12-14T01:07:42,489 (Thread-730) -   Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,489 (Thread-730) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,489 (Thread-730) -   Parameter 3: 'BCE78F106BCB6F912A5A10B55D182A53504AA6C2'
DEBUG 2018-12-14T01:07:42,489 (Thread-730) -   Parameter 4: ''
DEBUG 2018-12-14T01:07:42,489 (Thread-730) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,489 (Thread-730) - Done actual query (0ms): [UPDATE intrinsiclink
SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,489 (Startup thread) - Requested query: [UPDATE intrinsiclink SET
processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Actual query: [UPDATE intrinsiclink SET processid=?,isnew=?
WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-731) -   Parameter 0: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-731) -   Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,490 (Thread-731) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,490 (Thread-731) -   Parameter 3: '9DC7778DE94DBCD1BD2A7D7BAEFAB09919972A78'
DEBUG 2018-12-14T01:07:42,490 (Thread-731) -   Parameter 4: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-731) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-731) - Done actual query (0ms): [UPDATE intrinsiclink
SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Startup thread) - Requested query: [UPDATE intrinsiclink SET
processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-732) - Actual query: [UPDATE intrinsiclink SET processid=?,isnew=?
WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,490 (Thread-732) -   Parameter 0: ''
DEBUG 2018-12-14T01:07:42,490 (Thread-732) -   Parameter 1: 'E'
DEBUG 2018-12-14T01:07:42,490 (Thread-732) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,490 (Thread-732) -   Parameter 3: '00EFFBADCA2DC5EA9A03967A85DBCD87090B05B2'
DEBUG 2018-12-14T01:07:42,491 (Thread-732) -   Parameter 4: ''
DEBUG 2018-12-14T01:07:42,491 (Thread-732) -   Parameter 5: ''
DEBUG 2018-12-14T01:07:42,491 (Thread-732) - Done actual query (1ms): [UPDATE intrinsiclink
SET processid=?,isnew=? WHERE (jobid=? AND parentidhash=? AND linktype=? AND childidhash=?)]
DEBUG 2018-12-14T01:07:42,491 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,493 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,493 (Startup thread) - Beginning transaction of type 2
DEBUG 2018-12-14T01:07:42,493 (Startup thread) - Requested query: [UPDATE jobqueue SET isseed=?
WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,494 (Thread-733) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,494 (Thread-733) - Done actual query (0ms): [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,495 (Thread-734) - Actual query: [UPDATE jobqueue SET isseed=? WHERE
(isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,495 (Thread-734) -   Parameter 0: 'F'
DEBUG 2018-12-14T01:07:42,495 (Thread-734) -   Parameter 1: 'S'
DEBUG 2018-12-14T01:07:42,495 (Thread-734) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,496 (Thread-734) - Done actual query (2ms): [UPDATE jobqueue SET
isseed=? WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,496 (Startup thread) - Requested query: [UPDATE jobqueue SET isseed=?,seedingprocessid=NULL
WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,496 (Thread-735) - Actual query: [UPDATE jobqueue SET isseed=?,seedingprocessid=NULL
WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,496 (Thread-735) -   Parameter 0: 'S'
DEBUG 2018-12-14T01:07:42,496 (Thread-735) -   Parameter 1: 'S'
DEBUG 2018-12-14T01:07:42,497 (Thread-735) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,497 (Thread-735) - Done actual query (1ms): [UPDATE jobqueue SET
isseed=?,seedingprocessid=NULL WHERE (isseed=? AND jobid=?)]
DEBUG 2018-12-14T01:07:42,497 (Startup thread) - Marking for delete for job 1544121003866
all target document references matching 'isnew=B' that are targets of ( '' )
DEBUG 2018-12-14T01:07:42,497 (Startup thread) - Requested query: [UPDATE hopcount SET distance=?,deathmark=?
WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE (t0.jobid=? AND t0.childidhash=?) AND
EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE (t1.jobid=t0.jobid AND t1.linktype=t0.linktype
AND t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash) AND t1.isnew=?))]
DEBUG 2018-12-14T01:07:42,497 (Thread-736) - Actual query: [UPDATE hopcount SET distance=?,deathmark=?
WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE (t0.jobid=? AND t0.childidhash=?) AND
EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE (t1.jobid=t0.jobid AND t1.linktype=t0.linktype
AND t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash) AND t1.isnew=?))]
DEBUG 2018-12-14T01:07:42,497 (Thread-736) -   Parameter 0: '-1'
DEBUG 2018-12-14T01:07:42,497 (Thread-736) -   Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,497 (Thread-736) -   Parameter 2: '1544121003866'
DEBUG 2018-12-14T01:07:42,497 (Thread-736) -   Parameter 3: ''
DEBUG 2018-12-14T01:07:42,497 (Thread-736) -   Parameter 4: 'B'
DEBUG 2018-12-14T01:07:42,498 (Thread-736) - Done actual query (1ms): [UPDATE hopcount SET
distance=?,deathmark=? WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE (t0.jobid=?
AND t0.childidhash=?) AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE (t1.jobid=t0.jobid
AND t1.linktype=t0.linktype AND t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash)
AND t1.isnew=?))]
DEBUG 2018-12-14T01:07:42,498 (Startup thread) - Done marking for delete for job 1544121003866
DEBUG 2018-12-14T01:07:42,498 (Startup thread) - Requested query: [DELETE FROM intrinsiclink
WHERE (jobid=? AND childidhash=?) AND isnew=?]
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Actual query: [DELETE FROM intrinsiclink WHERE
(jobid=? AND childidhash=?) AND isnew=?]
DEBUG 2018-12-14T01:07:42,499 (Thread-737) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,499 (Thread-737) -   Parameter 1: ''
DEBUG 2018-12-14T01:07:42,499 (Thread-737) -   Parameter 2: 'B'
DEBUG 2018-12-14T01:07:42,499 (Thread-737) - Done actual query (0ms): [DELETE FROM intrinsiclink
WHERE (jobid=? AND childidhash=?) AND isnew=?]
DEBUG 2018-12-14T01:07:42,499 (Startup thread) - Requested query: [DELETE FROM hopdeletedeps
WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND deathmark=?))]
DEBUG 2018-12-14T01:07:42,499 (Thread-738) - Actual query: [DELETE FROM hopdeletedeps WHERE
ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND deathmark=?))]
DEBUG 2018-12-14T01:07:42,499 (Thread-738) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,499 (Thread-738) -   Parameter 1: 'D'
DEBUG 2018-12-14T01:07:42,500 (Thread-738) - Done actual query (1ms): [DELETE FROM hopdeletedeps
WHERE ownerid IN(SELECT id FROM hopcount WHERE (jobid=? AND deathmark=?))]
DEBUG 2018-12-14T01:07:42,500 (Startup thread) - Requested query: [UPDATE hopcount SET deathmark=?
WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,500 (Thread-739) - Actual query: [UPDATE hopcount SET deathmark=?
WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,500 (Thread-739) -   Parameter 0: 'Q'
DEBUG 2018-12-14T01:07:42,500 (Thread-739) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,501 (Thread-739) -   Parameter 2: 'D'
DEBUG 2018-12-14T01:07:42,501 (Thread-739) - Done actual query (1ms): [UPDATE hopcount SET
deathmark=? WHERE (jobid=? AND deathmark=?)]
DEBUG 2018-12-14T01:07:42,501 (Startup thread) - Done queueing for re-evaluation for 1544121003866
DEBUG 2018-12-14T01:07:42,501 (Startup thread) - Requested query: [UPDATE intrinsiclink SET
processid=NULL,isnew=? WHERE (jobid=? AND childidhash=?) AND isnew IN (?,?)]
DEBUG 2018-12-14T01:07:42,501 (Thread-740) - Actual query: [UPDATE intrinsiclink SET processid=NULL,isnew=?
WHERE (jobid=? AND childidhash=?) AND isnew IN (?,?)]
DEBUG 2018-12-14T01:07:42,501 (Thread-740) -   Parameter 0: 'B'
DEBUG 2018-12-14T01:07:42,501 (Thread-740) -   Parameter 1: '1544121003866'
DEBUG 2018-12-14T01:07:42,501 (Thread-740) -   Parameter 2: ''
DEBUG 2018-12-14T01:07:42,501 (Thread-740) -   Parameter 3: 'E'
DEBUG 2018-12-14T01:07:42,501 (Thread-740) -   Parameter 4: 'N'
DEBUG 2018-12-14T01:07:42,502 (Thread-740) - Done actual query (1ms): [UPDATE intrinsiclink
SET processid=NULL,isnew=? WHERE (jobid=? AND childidhash=?) AND isnew IN (?,?)]
DEBUG 2018-12-14T01:07:42,502 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,503 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,503 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,503 (Startup thread) - Requested query: [SELECT status,connectionname
FROM jobs WHERE id=? FOR UPDATE]
DEBUG 2018-12-14T01:07:42,504 (Thread-741) - Actual query: [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,504 (Thread-741) - Done actual query (0ms): [SET SCHEMA PUBLIC]
DEBUG 2018-12-14T01:07:42,504 (Thread-742) - Actual query: [SELECT status,connectionname FROM
jobs WHERE id=? FOR UPDATE]
DEBUG 2018-12-14T01:07:42,504 (Thread-742) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,505 (Thread-742) - Done actual query (1ms): [SELECT status,connectionname
FROM jobs WHERE id=? FOR UPDATE]
DEBUG 2018-12-14T01:07:42,505 (Startup thread) - Requested query: [SELECT transformationname
FROM jobpipelines WHERE (ownerid=? AND transformationname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,505 (Thread-743) - Actual query: [SELECT transformationname FROM
jobpipelines WHERE (ownerid=? AND transformationname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,505 (Thread-743) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,506 (Thread-743) - Done actual query (1ms): [SELECT transformationname
FROM jobpipelines WHERE (ownerid=? AND transformationname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,506 (Startup thread) - Requested query: [SELECT outputname FROM
jobpipelines WHERE (ownerid=? AND outputname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,506 (Thread-744) - Actual query: [SELECT outputname FROM jobpipelines
WHERE (ownerid=? AND outputname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,506 (Thread-744) -   Parameter 0: '1544121003866'
DEBUG 2018-12-14T01:07:42,507 (Thread-744) - Done actual query (1ms): [SELECT outputname FROM
jobpipelines WHERE (ownerid=? AND outputname IS NOT NULL)]
DEBUG 2018-12-14T01:07:42,507 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,507 (Startup thread) - Requested query: [SELECT classname FROM outputconnections
WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,507 (Thread-745) - Actual query: [SELECT classname FROM outputconnections
WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,507 (Thread-745) -   Parameter 0: 'Null'
DEBUG 2018-12-14T01:07:42,508 (Thread-745) - Done actual query (1ms): [SELECT classname FROM
outputconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,508 (Startup thread) - Requested query: [SELECT * FROM outputconnectors
WHERE classname=?]
DEBUG 2018-12-14T01:07:42,508 (Thread-746) - Actual query: [SELECT * FROM outputconnectors
WHERE classname=?]
DEBUG 2018-12-14T01:07:42,508 (Thread-746) -   Parameter 0: 'org.apache.manifoldcf.agents.output.nullconnector.NullConnector'
DEBUG 2018-12-14T01:07:42,509 (Thread-746) - Done actual query (1ms): [SELECT * FROM outputconnectors
WHERE classname=?]
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Beginning transaction of type 1
DEBUG 2018-12-14T01:07:42,509 (Startup thread) - Requested query: [SELECT classname FROM repoconnections
WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,509 (Thread-747) - Actual query: [SELECT classname FROM repoconnections
WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,509 (Thread-747) -   Parameter 0: 'Web'
DEBUG 2018-12-14T01:07:42,510 (Thread-747) - Done actual query (1ms): [SELECT classname FROM
repoconnections WHERE connectionname=?]
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Requested query: [SELECT * FROM connectors
WHERE classname=?]
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Committing transaction!
DEBUG 2018-12-14T01:07:42,510 (Startup thread) - Requested query: [UPDATE jobs SET failtime=NULL,seedingversion=?,processid=NULL,failcount=NULL,starttime=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,510 (Thread-748) - Actual query: [UPDATE jobs SET failtime=NULL,seedingversion=?,processid=NULL,failcount=NULL,starttime=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,510 (Thread-748) -   Parameter 0: ''
DEBUG 2018-12-14T01:07:42,510 (Thread-748) -   Parameter 1: '1544767662340'
DEBUG 2018-12-14T01:07:42,510 (Thread-748) -   Parameter 2: 'A'
DEBUG 2018-12-14T01:07:42,510 (Thread-748) -   Parameter 3: '1544121003866'
DEBUG 2018-12-14T01:07:42,511 (Thread-748) - Done actual query (1ms): [UPDATE jobs SET failtime=NULL,seedingversion=?,processid=NULL,failcount=NULL,starttime=?,status=?
WHERE id=?]
DEBUG 2018-12-14T01:07:42,511 (Startup thread) - Ending transaction
DEBUG 2018-12-14T01:07:42,511 (Startup thread) - Committing transaction!
{code}



> Documents unreachable due to hopcount are not considered unreachable on cleanup pass
> ------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1562
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1562
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Elastic Search connector, Web connector
>    Affects Versions: ManifoldCF 2.11
>         Environment: Manifoldcf 2.11
> Elasticsearch 6.3.2
> Web inputconnector
> elastic outputconnecotr
> Job crawls website input and outputs content to elastic
>            Reporter: Tim Steenbeke
>            Assignee: Karl Wright
>            Priority: Critical
>              Labels: starter
>         Attachments: manifoldcf.log.cleanup, manifoldcf.log.init, manifoldcf.log.reduced
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> My documents aren't removed from ElasticSearch index after rerunning the changed seeds
> I update my job to change the seedmap and rerun it or use the schedualer to keep it runneng
even after updating it.
> After the rerun the unreachable documents don't get deleted.
> It only adds doucments when they can be reached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message