manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shigeki Kobayashi <shigeki.kobayas...@g.softbank.co.jp>
Subject Re: Web crawling causes Socket Timeout after Database Exception
Date Tue, 27 Nov 2012 05:50:10 GMT
Hi Karl,

According to INNODB STATUS in MySQL, while crawling web, the following
DEADLOCK occurred.
A few minutes later, database exception occurred in MCF.
So do you think probably this DEADLOCK could cause the exception?

I do not know the error code from MySQL yet, but maybe I could obtain it if
you could let me
know what code in what file should be added in order to output the error
code into manifold.log


------------------------------------------------------------------------------
INNODB STATUS:
=====================================
121127 11:17:49 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 60 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 401163 1_second, 401162 sleeps, 40049 10_second,
674 background, 674 flush
srv_master_thread log flush and writes: 401182
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 7319, signal count 6842
Mutex spin waits 3702, rounds 111120, OS waits 3626
RW-shared spins 2189, rounds 63516, OS waits 1767
RW-excl spins 255, rounds 57147, OS waits 1897
Spin rounds per wait: 30.02 mutex, 29.02 RW-shared, 224.11 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
121122 19:31:55
*** (1) TRANSACTION:
TRANSACTION 3021A0, ACTIVE 32 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 64 lock struct(s), heap size 14776, 110 row lock(s), undo log
entries 51
MySQL thread id 24, OS thread handle 0x7ff8ffe06700, query id 41385
10.249.23.9 manifoldcf Sending data
SELECT parentidhash,linktype,distance FROM hopcount WHERE
jobid=1351139121625 AND parentidhash IN
('A0ED08F9D45547FF54B72869FE5E7C3C5B0E910A','F5E2F6C6B43FB5D030C4F0AE8E22AD07536475A8','C0856A1AFF55F7BB20BCAE317E18F588EEFB806D','019253D99FCB265A20A3CFF11D0443937FE2D4D0','21A979F9BB9120F747B0B605EDABA71EB364A584','A8B5B7245D0810584B764470B42CFDF71C33A7E5','6FE272988943D3BD64E285951A1A6739011FC15E','1E1CA954A3E31BFC28FFE1BE70757408341CDB6A','8DAE8B4734A30FE2D346EEBD1CCC3A16468F7B7B','028CD3E7FF7F493E3EC3980FF303DB05DC42404E','924E0608A5C4505C9272A69B8C1F82C7B883A11F','13F6402C96E0979EF5F17338DFF96BD9912125D9','43174E34AA07C34237D622A43A82AFE3825C3870','32892282A6866BD181BDA0BA85801192370C84F3','0311197289655163E1452E90D43A5D96D9A4E751','178C8BE84AEDC9F362CE3A2CC2702F6C2CD9CBA1','7CF15B193B3BDA097BEB437272FC5E413B86B63D')
AND linktype IN ('link','redirect')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 2449 n bits 192 index `PRIMARY` of table
`manifoldcf`.`hopcount` trx id 3021A0 lock mode S locks rec but not gap
waiting
Record lock, heap no 28 PHYSICAL RECORD: n_fields 8; compact format; info
bits 0
 0: len 8; hex 8000013b261c6d8a; asc    ;& m ;;
 1: len 6; hex 0000002f4e47; asc    /NG;;
 2: len 7; hex 900000021b06ec; asc        ;;
 3: len 1; hex 4e; asc N;;
 4: len 30; hex
384441453842343733344133304645324433343645454244314343433341; asc
8DAE8B4734A30FE2D346EEBD1CCC3A; (total 40 bytes);
 5: len 8; hex 8000000000000001; asc         ;;
 6: len 8; hex 8000013a962ad9d9; asc    : *  ;;
 7: len 4; hex 6c696e6b; asc link;;

*** (2) TRANSACTION:
TRANSACTION 302208, ACTIVE 3 sec fetching rows
mysql tables in use 3, locked 3
1436 lock struct(s), heap size 145848, 122906 row lock(s)
MySQL thread id 39, OS thread handle 0x7ff8ffa37700, query id 40699
10.249.23.9 manifoldcf preparing
UPDATE hopcount SET deathmark='D',distance=-1 WHERE id IN(SELECT ownerid
FROM hopdeletedeps t0 WHERE t0.jobid=1351139121625 AND
t0.childidhash='D573BDC6D59C7A7CC2862646322F69EA5574C36D' AND EXISTS(SELECT
'x' FROM intrinsiclink t1 WHERE t1.jobid=t0.jobid AND
t1.linktype=t0.linktype AND t1.parentidhash=t0.parentidhash AND
t1.childidhash=t0.childidhash AND t1.isnew='B'))
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 2449 n bits 192 index `PRIMARY` of table
`manifoldcf`.`hopcount` trx id 302208 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info
bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

...
...
...

*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 38375F
Purge done for trx's n:o < 3024F7 undo n:o < 0
History list length 652
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 110, OS thread handle 0x7ff914113700, query id 1436936
localhost root
SHOW ENGINE INNODB STATUS
---TRANSACTION 0, not started
MySQL thread id 106, OS thread handle 0x7ff9035b3700, query id 1435785
localhost root
---TRANSACTION 38375E, not started
MySQL thread id 99, OS thread handle 0x7ff8ff72b700, query id 1436934
10.249.23.9 manifoldcf
--------
...
------------------------------------------------------------------------------


Likewise, file crawling using Windows shares faced similar matter. DEADLOCK
occured in MySQL and
Database Exception occured in MCF as well:

------------------------------------------------------------------------------

| InnoDB |      |
=====================================
121126 16:05:21 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 48 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 327427 1_second, 327300 sleeps, 32438 10_second,
3544 background, 3544 flush
srv_master_thread log flush and writes: 327670
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 1808090, signal count 2140762
Mutex spin waits 18194682, rounds 103331992, OS waits 842070
RW-shared spins 1311114, rounds 25796436, OS waits 457767
RW-excl spins 577964, rounds 15904805, OS waits 333210
Spin rounds per wait: 5.68 mutex, 19.68 RW-shared, 27.52 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
121122 23:38:46
*** (1) TRANSACTION:
TRANSACTION 674749, ACTIVE 7 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 4 lock struct(s), heap size 1248, 3 row lock(s), undo log entries
1
MySQL thread id 99, OS thread handle 0x7f7d4a356700, query id 23942404
localhost 127.0.0.1 manifoldcf update
INSERT INTO jobqueue
(docpriority,id,priorityset,docid,status,dochash,checktime,checkaction,jobid)
VALUES
(13.830866056523654,1353595119848,1353595119385,'smb://xxx/xxx','P','88517951DB2E0666151E7B5308C9FDCB16F062AD',0,'R',1353575409046)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 221894 n bits 208 index `I1352346865065` of
table `manifoldcf`.`jobqueue` trx id 674749 lock_mode X locks gap before
rec insert intention waiting
Record lock, heap no 134 PHYSICAL RECORD: n_fields 3; compact format; info
bits 0
 0: len 30; hex
383835313838444433453134444134354242384531383433424330393444; asc
885188DD3E14DA45BB8E1843BC094D; (total 40 bytes);
 1: len 8; hex 8000013b2761a596; asc    ;'a  ;;
 2: len 8; hex 8000013b287bd5c9; asc    ;({  ;;

*** (2) TRANSACTION:
TRANSACTION 6740DF, ACTIVE 9 sec fetching rows
mysql tables in use 5, locked 5
23571 lock struct(s), heap size 2439608, 1058037 row lock(s)
MySQL thread id 45, OS thread handle 0x7f7d21231700, query id 23937374
localhost 127.0.0.1 manifoldcf Sending data
SELECT t0.id,t0.jobid,t0.dochash,t0.docid,t0.status,t0.failtime,t0.failcount,t0.priorityset
FROM jobqueue t0 WHERE t0.status IN ('P','G') AND t0.checkaction='R' AND
t0.checktime<=1353595117855 AND EXISTS(SELECT 'x' FROM jobs t1 WHERE
t1.status IN ('A','a') AND t1.id=t0.jobid AND t1.priority=5) AND NOT
EXISTS(SELECT 'x' FROM jobqueue t2 WHERE t2.dochash=t0.dochash AND
t2.status IN ('A','F','a','f','D','d') AND t2.jobid!=t0.jobid) AND NOT
EXISTS(SELECT 'x' FROM prereqevents t3,events t4 WHERE t0.id=t3.owner AND
t3.eventname=t4.name) ORDER BY t0.docpriority ASC,t0.status
ASC,t0.checkaction ASC,t0.checktime ASC LIMIT 1200
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 221894 n bits 208 index `I1352346865065` of
table `manifoldcf`.`jobqueue` trx id 6740DF lock mode S locks gap before rec
Record lock, heap no 8 PHYSICAL RECORD: n_fields 3; compact format; info
bits 0
 0: len 30; hex
383834464239393738383632333242323331353041343031303337424444; asc
884FB997886232B23150A401037BDD; (total 40 bytes);
 1: len 8; hex 8000013b2761a596; asc    ;'a  ;;
 2: len 8; hex 8000013b27c4823b; asc    ;'  ;;;

...
...
...

*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 6ACDF6
Purge done for trx's n:o < 6752D1 undo n:o < 0
History list length 485
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 5505, OS thread handle 0x7f7d210ec700, query id 25071245
localhost root
SHOW ENGINE INNODB STATUS
---TRANSACTION 6ACDF5, not started
MySQL thread id 99, OS thread handle 0x7f7d4a356700, query id 25071244
localhost 127.0.0.1 manifoldcf
...
------------------------------------------------------------------------------




Regards,

Shigeki


2012/10/19 Shigeki Kobayashi <shigeki.kobayashi3@g.softbank.co.jp>

> Due to the error, I had to downgrade to a lower version so I haven't found
> the MySQL error code yet.
>
> I installed MCF1.0 in a different environment where crawlable contents are
> different from the above environment.
> I could not reproduce the Database exception but socket timeout occurred
> In the same environment, I ran MCF0.6 and it completed crawling without
> socket timeout.
> Like you said, socket timeout seems to be a different problem from the
> Database exception .
>
> 2012/10/18 Karl Wright <daddywri@gmail.com>
>
>> So, what was the resolution of this problem?  Any news?
>> Karl
>>
>> On Thu, Oct 11, 2012 at 2:28 AM, Karl Wright <daddywri@gmail.com> wrote:
>> > The only change is that the MySQL driver now performs ANALYZE
>> > operations on the fly in order to keep the database operating at high
>> > efficiency.  This is CONNECTORS-510.  It is possible that, on a large
>> > database table, these operations will cause others to wait long enough
>> > so that their timeout is exceeded.  Such an event does not take place
>> > while the load tests run, however.  If you want to turn off the
>> > analyze operation, you can do that by setting a per-table property to
>> > override the analyze default of 10000 operations:
>> >
>> > analyzeThreshold =
>> >
>> ManifoldCF.getIntProperty("org.apache.manifold.db.mysql.analyze."+tableName,10000);
>> >
>> > The table in question is "jobqueue".  If you set this value to
>> > something like 1000000000 and you still see MySQL timeouts, then this
>> > new code is not the problem.  And, like I said, the best solution is
>> > to recognize the error and retry, but first I would need the error
>> > code.  Adding an appropriate output of sqlState around line 123 of
>> >
>> framework/core/src/main/java/org/apache/manifoldcf/core/database/DBInterfaceMySQL.java
>> > would allow us to see what code to catch, when it happened again.
>> >
>> > For the Web connector, the only modifications have been in regards to
>> > how it handles 500 errors, which now correctly code to avoid an
>> > IndexExceptionOutOfBounds exception.  This has nothing to do with
>> > socket exceptions, which are caused for external reasons only.
>> >
>> > Karl
>> >
>> >
>> > On Wed, Oct 10, 2012 at 10:32 PM, Shigeki Kobayashi
>> > <shigeki.kobayashi3@g.softbank.co.jp> wrote:
>> >> Hi Karl,
>> >>
>> >>
>> >> I was comparing version 1.0 with old trunk based on version 0.6
>> implementing
>> >> CONNECTORS-501(
>> >> Medium-scale web crawl with hopcount-based filtering fails to find
>> correct
>> >> number of documents).
>> >>
>> >> Running each version with the same MySQL setting and the same
>> throttling,
>> >> somehow the version 1.0 hangs with the error.
>> >> Since the old trunk completes crawling, I wonder if something has
>> changed.
>> >>
>> >> Just to make sure I will recheck if there are any wrong settings in
>> MCF.
>> >>
>> >> Thanks.
>> >>
>> >> Regards,
>> >>
>> >> Shigeki
>> >>
>> >> 2012/10/10 Karl Wright <daddywri@gmail.com>
>> >>>
>> >>> Hi Shigeki,
>> >>>
>> >>> The socket timeout exception is only a warning.  It means that some
>> >>> site you are crawling did not accept a socket connection within the
>> >>> allowed time (5 minutes I think).  The Web Connector will retry the
>> >>> connection a few times, and if it is still rejected, it will
>> >>> eventually give up on that page.  One thing you want to check, though,
>> >>> is that you are using proper throttling, because if you aren't then
>> >>> one cause of this problem is that the webmaster of the site you are
>> >>> trying to crawl may have blocked you from accessing it.
>> >>>
>> >>> The database exception is more problematic.  It means that MySQL
>> >>> thinks it took too long for a specific transaction to complete, and
>> >>> the database aborted the transaction due to a timeout.  There are two
>> >>> ways of dealing with this issue.  One way is to modify your MySQL
>> >>> configuration to increase the transaction timeout value to some high
>> >>> number.  The second way is to modify ManifoldCF to recognize the
>> >>> timeout error specifically, and cause a retry.  But in order to do the
>> >>> latter, I would need to know what SQL error code MySQL returns for
>> >>> this situation, which will mean we either need to look it up (if we
>> >>> can), or modify a ManifoldCF instance to log it when this problem
>> >>> occurs.
>> >>>
>> >>> Please let me know how you would like to proceed.
>> >>>
>> >>> Karl
>> >>>
>> >>> On Wed, Oct 10, 2012 at 3:51 AM, Shigeki Kobayashi
>> >>> <shigeki.kobayashi3@g.softbank.co.jp> wrote:
>> >>> >
>> >>> > Hi
>> >>> >
>> >>> > I am having a trouble with crawling web using MCF1.0.
>> >>> > I run MCF with MySQL 5.5 and Tomcat 6.0.
>> >>> > It should keep crawling contents, but MCF prints the following
>> Database
>> >>> > exception log, then hangs.
>> >>> > After DB Exception, Socket Time Exception occurs.
>> >>> >
>> >>> > Anyone has faced this problem?
>> >>> >
>> >>> > --Database Exception log:
>> >>> >
>> >>> > ERROR 2012-10-10 16:11:05,787 (Worker thread '42') - Worker thread
>> >>> > aborting
>> >>> > and restarting due to database connection reset: Database exception:
>> >>> > Exception doing query: Lock wait timeout exceeded; try restarting
>> >>> > transaction
>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>> >>> > exception: Exception doing query: Lock wait timeout exceeded; try
>> >>> > restarting
>> >>> > transaction
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessAcivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
>> >>> > Caused by: java.sql.SQLException: Lock wait timeout exceeded; try
>> >>> > restarting
>> >>> > transaction
>> >>> >         at
>> >>> > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
>> >>> >         at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
>> >>> >         at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
>> >>> >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
>> >>> >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
>> >>> >         at
>> >>> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
>> >>> >         at
>> >>> >
>> >>> >
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
>> >>> >         at
>> >>> >
>> >>> >
>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
>> >>> >         at
>> >>> >
>> org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>> >>> > ERROR 2012-10-10 16:11:06,799 (Worker thread '9') - Worker thread
>> >>> > aborting
>> >>> > and restarting due to database connection reset: Database exception:
>> >>> > Exception doing query: Lock wait timeout exceeded; try restarting
>> >>> > transaction
>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>> >>> > exception: Exception doing query: Lock wait timeout exceeded; try
>> >>> > restarting
>> >>> > transaction
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1863)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:554)
>> >>> > Caused by: java.sql.SQLException: Lock wait timeout exceeded; try
>> >>> > restarting
>> >>> > transaction
>> >>> >         at
>> >>> > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
>> >>> >         at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
>> >>> >         at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
>> >>> >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
>> >>> >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
>> >>> >         at
>> >>> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
>> >>> >         at
>> >>> >
>> >>> >
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
>> >>> >         at
>> >>> >
>> >>> >
>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
>> >>> >         at
>> >>> >
>> org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>> >>> >
>> >>> >
>> >>> >
>> >>> > ---- Socket Timeout:
>> >>> >
>> >>> >
>> >>> > DEBUG 2012-10-10 16:16:27,256 (Worker thread '49') - Socket timeout
>> >>> > exception trying to close connection: Read timed out
>> >>> > java.net.SocketTimeoutException: Read timed out
>> >>> >         at java.net.SocketInputStream.socketRead0(Native Method)
>> >>> >         at
>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>> >>> >         at
>> >>> > java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> >>> >         at
>> >>> > java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>> >>> >         at
>> >>> > java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> >>> >         at
>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown
>> >>> > Source)
>> >>> >         at
>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown
>> >>> > Source)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.commons.httpclient.ChunkedInputStream.exhaustInputStream(Unknown
>> >>> > Source)
>> >>> >         at
>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.close(Unknown
>> >>> > Source)
>> >>> >         at
>> java.io.FilterInputStream.close(FilterInputStream.java:155)
>> >>> >         at
>> >>> >
>> org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(Unknown
>> >>> > Source)
>> >>> >         at
>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.close(Unknown
>> >>> > Source)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.close(ThrottledFetcher.java:2082)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:176)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:745)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>> >>> >  INFO 2012-10-10 16:16:27,273 (Worker thread '49') - WEB: FETCH
>> >>> >
>> >>> > URL|
>> http://xxxxxx/...|1349852786744+600514|-104|4125|org.apache.manifoldcf.core.interfaces.ManifoldCFException|<http://xxxxxx/...%7C1349852786744+600514%7C-104%7C4125%7Corg.apache.manifoldcf.core.interfaces.ManifoldCFException%7C>
>> >>> > Interrupted: Socket timeout: Read timed out
>> >>> > DEBUG 2012-10-10 16:16:27,273 (Worker thread '49') - WEB: Fetch
>> >>> > exception
>> >>> > for 'http://xxxxxx/...'
>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>> Interrupted:
>> >>> > Socket timeout: Read timed out
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.noteInterrupted(ThrottledFetcher.java:1818)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:797)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>> >>> > Caused by:
>> org.apache.manifoldcf.agents.interfaces.ServiceInterruption:
>> >>> > Socket timeout: Read timed out
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:745)
>> >>> >         ... 1 more
>> >>> > Caused by: java.net.SocketTimeoutException: Read timed out
>> >>> >         at java.net.SocketInputStream.socketRead0(Native Method)
>> >>> >         at
>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>> >>> >         at
>> >>> > java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>> >>> >         at
>> >>> > java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> >>> >         at
>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown
>> >>> > Source)
>> >>> >         at
>> java.io.FilterInputStream.read(FilterInputStream.java:116)
>> >>> >         at
>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.read(Unknown
>> >>> > Source)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2012)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:1976)
>> >>> >         at
>> >>> >
>> >>> >
>> org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:95)
>> >>> >         ... 2 more
>> >>> >  WARN 2012-10-10 16:16:27,274 (Worker thread '49') - Pre-ingest
>> service
>> >>> > interruption reported for job 1349774325961 connection 'WEB': Socket
>> >>> > timeout: Read timed out
>> >>> >
>> >>> >
>> >>> >
>> >>> > Regards,
>> >>> >
>> >>> > Shigeki
>> >>
>> >>
>> >>
>> >>
>>
>
>
>
>

Mime
View raw message