manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Web crawling causes Socket Timeout after Database Exception
Date Tue, 27 Nov 2012 10:31:22 GMT
Hi Shigeki,

Deadlocks are a fact of life in a very multithreaded application.
They are supposed to be caught by ManifoldCF, and the transactions
retried.  I can believe, though, that MySQL might set different
sqlexception status codes for different kinds of deadlock - if instead
of a sqlexception with a deadlock code, MySQL sometimes just drops the
JDBC connection, that might explain the problem.

Can you refresh my memory and please send the ManifoldCF log part that
includes the socket timeout exception?  I can then see if it is coming
from the same place.

Karl

On Tue, Nov 27, 2012 at 12:50 AM, Shigeki Kobayashi
<shigeki.kobayashi3@g.softbank.co.jp> wrote:
> Hi Karl,
>
> According to INNODB STATUS in MySQL, while crawling web, the following
> DEADLOCK occurred.
> A few minutes later, database exception occurred in MCF.
> So do you think probably this DEADLOCK could cause the exception?
>
> I do not know the error code from MySQL yet, but maybe I could obtain it if
> you could let me
> know what code in what file should be added in order to output the error
> code into manifold.log
>
>
> ------------------------------------------------------------------------------
> INNODB STATUS:
> =====================================
> 121127 11:17:49 INNODB MONITOR OUTPUT
> =====================================
> Per second averages calculated from the last 60 seconds
> -----------------
> BACKGROUND THREAD
> -----------------
> srv_master_thread loops: 401163 1_second, 401162 sleeps, 40049 10_second,
> 674 background, 674 flush
> srv_master_thread log flush and writes: 401182
> ----------
> SEMAPHORES
> ----------
> OS WAIT ARRAY INFO: reservation count 7319, signal count 6842
> Mutex spin waits 3702, rounds 111120, OS waits 3626
> RW-shared spins 2189, rounds 63516, OS waits 1767
> RW-excl spins 255, rounds 57147, OS waits 1897
> Spin rounds per wait: 30.02 mutex, 29.02 RW-shared, 224.11 RW-excl
> ------------------------
> LATEST DETECTED DEADLOCK
> ------------------------
> 121122 19:31:55
> *** (1) TRANSACTION:
> TRANSACTION 3021A0, ACTIVE 32 sec starting index read
> mysql tables in use 1, locked 1
> LOCK WAIT 64 lock struct(s), heap size 14776, 110 row lock(s), undo log
> entries 51
> MySQL thread id 24, OS thread handle 0x7ff8ffe06700, query id 41385
> 10.249.23.9 manifoldcf Sending data
> SELECT parentidhash,linktype,distance FROM hopcount WHERE
> jobid=1351139121625 AND parentidhash IN
> ('A0ED08F9D45547FF54B72869FE5E7C3C5B0E910A','F5E2F6C6B43FB5D030C4F0AE8E22AD07536475A8','C0856A1AFF55F7BB20BCAE317E18F588EEFB806D','019253D99FCB265A20A3CFF11D0443937FE2D4D0','21A979F9BB9120F747B0B605EDABA71EB364A584','A8B5B7245D0810584B764470B42CFDF71C33A7E5','6FE272988943D3BD64E285951A1A6739011FC15E','1E1CA954A3E31BFC28FFE1BE70757408341CDB6A','8DAE8B4734A30FE2D346EEBD1CCC3A16468F7B7B','028CD3E7FF7F493E3EC3980FF303DB05DC42404E','924E0608A5C4505C9272A69B8C1F82C7B883A11F','13F6402C96E0979EF5F17338DFF96BD9912125D9','43174E34AA07C34237D622A43A82AFE3825C3870','32892282A6866BD181BDA0BA85801192370C84F3','0311197289655163E1452E90D43A5D96D9A4E751','178C8BE84AEDC9F362CE3A2CC2702F6C2CD9CBA1','7CF15B193B3BDA097BEB437272FC5E413B86B63D')
> AND linktype IN ('link','redirect')
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 0 page no 2449 n bits 192 index `PRIMARY` of table
> `manifoldcf`.`hopcount` trx id 3021A0 lock mode S locks rec but not gap
> waiting
> Record lock, heap no 28 PHYSICAL RECORD: n_fields 8; compact format; info
> bits 0
>  0: len 8; hex 8000013b261c6d8a; asc    ;& m ;;
>  1: len 6; hex 0000002f4e47; asc    /NG;;
>  2: len 7; hex 900000021b06ec; asc        ;;
>  3: len 1; hex 4e; asc N;;
>  4: len 30; hex
> 384441453842343733344133304645324433343645454244314343433341; asc
> 8DAE8B4734A30FE2D346EEBD1CCC3A; (total 40 bytes);
>  5: len 8; hex 8000000000000001; asc         ;;
>  6: len 8; hex 8000013a962ad9d9; asc    : *  ;;
>  7: len 4; hex 6c696e6b; asc link;;
>
> *** (2) TRANSACTION:
> TRANSACTION 302208, ACTIVE 3 sec fetching rows
> mysql tables in use 3, locked 3
> 1436 lock struct(s), heap size 145848, 122906 row lock(s)
> MySQL thread id 39, OS thread handle 0x7ff8ffa37700, query id 40699
> 10.249.23.9 manifoldcf preparing
> UPDATE hopcount SET deathmark='D',distance=-1 WHERE id IN(SELECT ownerid
> FROM hopdeletedeps t0 WHERE t0.jobid=1351139121625 AND
> t0.childidhash='D573BDC6D59C7A7CC2862646322F69EA5574C36D' AND EXISTS(SELECT
> 'x' FROM intrinsiclink t1 WHERE t1.jobid=t0.jobid AND
> t1.linktype=t0.linktype AND t1.parentidhash=t0.parentidhash AND
> t1.childidhash=t0.childidhash AND t1.isnew='B'))
> *** (2) HOLDS THE LOCK(S):
> RECORD LOCKS space id 0 page no 2449 n bits 192 index `PRIMARY` of table
> `manifoldcf`.`hopcount` trx id 302208 lock_mode X
> Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info
> bits 0
>  0: len 8; hex 73757072656d756d; asc supremum;;
>
> ...
> ...
> ...
>
> *** WE ROLL BACK TRANSACTION (1)
> ------------
> TRANSACTIONS
> ------------
> Trx id counter 38375F
> Purge done for trx's n:o < 3024F7 undo n:o < 0
> History list length 652
> LIST OF TRANSACTIONS FOR EACH SESSION:
> ---TRANSACTION 0, not started
> MySQL thread id 110, OS thread handle 0x7ff914113700, query id 1436936
> localhost root
> SHOW ENGINE INNODB STATUS
> ---TRANSACTION 0, not started
> MySQL thread id 106, OS thread handle 0x7ff9035b3700, query id 1435785
> localhost root
> ---TRANSACTION 38375E, not started
> MySQL thread id 99, OS thread handle 0x7ff8ff72b700, query id 1436934
> 10.249.23.9 manifoldcf
> --------
> ...
> ------------------------------------------------------------------------------
>
>
> Likewise, file crawling using Windows shares faced similar matter. DEADLOCK
> occured in MySQL and
> Database Exception occured in MCF as well:
>
> ------------------------------------------------------------------------------
>
> | InnoDB |      |
> =====================================
> 121126 16:05:21 INNODB MONITOR OUTPUT
> =====================================
> Per second averages calculated from the last 48 seconds
> -----------------
> BACKGROUND THREAD
> -----------------
> srv_master_thread loops: 327427 1_second, 327300 sleeps, 32438 10_second,
> 3544 background, 3544 flush
> srv_master_thread log flush and writes: 327670
> ----------
> SEMAPHORES
> ----------
> OS WAIT ARRAY INFO: reservation count 1808090, signal count 2140762
> Mutex spin waits 18194682, rounds 103331992, OS waits 842070
> RW-shared spins 1311114, rounds 25796436, OS waits 457767
> RW-excl spins 577964, rounds 15904805, OS waits 333210
> Spin rounds per wait: 5.68 mutex, 19.68 RW-shared, 27.52 RW-excl
> ------------------------
> LATEST DETECTED DEADLOCK
> ------------------------
> 121122 23:38:46
> *** (1) TRANSACTION:
> TRANSACTION 674749, ACTIVE 7 sec inserting
> mysql tables in use 1, locked 1
> LOCK WAIT 4 lock struct(s), heap size 1248, 3 row lock(s), undo log entries
> 1
> MySQL thread id 99, OS thread handle 0x7f7d4a356700, query id 23942404
> localhost 127.0.0.1 manifoldcf update
> INSERT INTO jobqueue
> (docpriority,id,priorityset,docid,status,dochash,checktime,checkaction,jobid)
> VALUES
> (13.830866056523654,1353595119848,1353595119385,'smb://xxx/xxx','P','88517951DB2E0666151E7B5308C9FDCB16F062AD',0,'R',1353575409046)
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 0 page no 221894 n bits 208 index `I1352346865065` of
> table `manifoldcf`.`jobqueue` trx id 674749 lock_mode X locks gap before rec
> insert intention waiting
> Record lock, heap no 134 PHYSICAL RECORD: n_fields 3; compact format; info
> bits 0
>  0: len 30; hex
> 383835313838444433453134444134354242384531383433424330393444; asc
> 885188DD3E14DA45BB8E1843BC094D; (total 40 bytes);
>  1: len 8; hex 8000013b2761a596; asc    ;'a  ;;
>  2: len 8; hex 8000013b287bd5c9; asc    ;({  ;;
>
> *** (2) TRANSACTION:
> TRANSACTION 6740DF, ACTIVE 9 sec fetching rows
> mysql tables in use 5, locked 5
> 23571 lock struct(s), heap size 2439608, 1058037 row lock(s)
> MySQL thread id 45, OS thread handle 0x7f7d21231700, query id 23937374
> localhost 127.0.0.1 manifoldcf Sending data
> SELECT
> t0.id,t0.jobid,t0.dochash,t0.docid,t0.status,t0.failtime,t0.failcount,t0.priorityset
> FROM jobqueue t0 WHERE t0.status IN ('P','G') AND t0.checkaction='R' AND
> t0.checktime<=1353595117855 AND EXISTS(SELECT 'x' FROM jobs t1 WHERE
> t1.status IN ('A','a') AND t1.id=t0.jobid AND t1.priority=5) AND NOT
> EXISTS(SELECT 'x' FROM jobqueue t2 WHERE t2.dochash=t0.dochash AND t2.status
> IN ('A','F','a','f','D','d') AND t2.jobid!=t0.jobid) AND NOT EXISTS(SELECT
> 'x' FROM prereqevents t3,events t4 WHERE t0.id=t3.owner AND
> t3.eventname=t4.name) ORDER BY t0.docpriority ASC,t0.status
> ASC,t0.checkaction ASC,t0.checktime ASC LIMIT 1200
> *** (2) HOLDS THE LOCK(S):
> RECORD LOCKS space id 0 page no 221894 n bits 208 index `I1352346865065` of
> table `manifoldcf`.`jobqueue` trx id 6740DF lock mode S locks gap before rec
> Record lock, heap no 8 PHYSICAL RECORD: n_fields 3; compact format; info
> bits 0
>  0: len 30; hex
> 383834464239393738383632333242323331353041343031303337424444; asc
> 884FB997886232B23150A401037BDD; (total 40 bytes);
>  1: len 8; hex 8000013b2761a596; asc    ;'a  ;;
>  2: len 8; hex 8000013b27c4823b; asc    ;'  ;;;
>
> ...
> ...
> ...
>
> *** WE ROLL BACK TRANSACTION (1)
> ------------
> TRANSACTIONS
> ------------
> Trx id counter 6ACDF6
> Purge done for trx's n:o < 6752D1 undo n:o < 0
> History list length 485
> LIST OF TRANSACTIONS FOR EACH SESSION:
> ---TRANSACTION 0, not started
> MySQL thread id 5505, OS thread handle 0x7f7d210ec700, query id 25071245
> localhost root
> SHOW ENGINE INNODB STATUS
> ---TRANSACTION 6ACDF5, not started
> MySQL thread id 99, OS thread handle 0x7f7d4a356700, query id 25071244
> localhost 127.0.0.1 manifoldcf
> ...
> ------------------------------------------------------------------------------
>
>
>
>
> Regards,
>
> Shigeki
>
>
> 2012/10/19 Shigeki Kobayashi <shigeki.kobayashi3@g.softbank.co.jp>
>>
>> Due to the error, I had to downgrade to a lower version so I haven't found
>> the MySQL error code yet.
>>
>> I installed MCF1.0 in a different environment where crawlable contents are
>> different from the above environment.
>> I could not reproduce the Database exception but socket timeout occurred
>> In the same environment, I ran MCF0.6 and it completed crawling without
>> socket timeout.
>> Like you said, socket timeout seems to be a different problem from the
>> Database exception .
>>
>> 2012/10/18 Karl Wright <daddywri@gmail.com>
>>>
>>> So, what was the resolution of this problem?  Any news?
>>> Karl
>>>
>>> On Thu, Oct 11, 2012 at 2:28 AM, Karl Wright <daddywri@gmail.com> wrote:
>>> > The only change is that the MySQL driver now performs ANALYZE
>>> > operations on the fly in order to keep the database operating at high
>>> > efficiency.  This is CONNECTORS-510.  It is possible that, on a large
>>> > database table, these operations will cause others to wait long enough
>>> > so that their timeout is exceeded.  Such an event does not take place
>>> > while the load tests run, however.  If you want to turn off the
>>> > analyze operation, you can do that by setting a per-table property to
>>> > override the analyze default of 10000 operations:
>>> >
>>> > analyzeThreshold =
>>> >
>>> > ManifoldCF.getIntProperty("org.apache.manifold.db.mysql.analyze."+tableName,10000);
>>> >
>>> > The table in question is "jobqueue".  If you set this value to
>>> > something like 1000000000 and you still see MySQL timeouts, then this
>>> > new code is not the problem.  And, like I said, the best solution is
>>> > to recognize the error and retry, but first I would need the error
>>> > code.  Adding an appropriate output of sqlState around line 123 of
>>> >
>>> > framework/core/src/main/java/org/apache/manifoldcf/core/database/DBInterfaceMySQL.java
>>> > would allow us to see what code to catch, when it happened again.
>>> >
>>> > For the Web connector, the only modifications have been in regards to
>>> > how it handles 500 errors, which now correctly code to avoid an
>>> > IndexExceptionOutOfBounds exception.  This has nothing to do with
>>> > socket exceptions, which are caused for external reasons only.
>>> >
>>> > Karl
>>> >
>>> >
>>> > On Wed, Oct 10, 2012 at 10:32 PM, Shigeki Kobayashi
>>> > <shigeki.kobayashi3@g.softbank.co.jp> wrote:
>>> >> Hi Karl,
>>> >>
>>> >>
>>> >> I was comparing version 1.0 with old trunk based on version 0.6
>>> >> implementing
>>> >> CONNECTORS-501(
>>> >> Medium-scale web crawl with hopcount-based filtering fails to find
>>> >> correct
>>> >> number of documents).
>>> >>
>>> >> Running each version with the same MySQL setting and the same
>>> >> throttling,
>>> >> somehow the version 1.0 hangs with the error.
>>> >> Since the old trunk completes crawling, I wonder if something has
>>> >> changed.
>>> >>
>>> >> Just to make sure I will recheck if there are any wrong settings in
>>> >> MCF.
>>> >>
>>> >> Thanks.
>>> >>
>>> >> Regards,
>>> >>
>>> >> Shigeki
>>> >>
>>> >> 2012/10/10 Karl Wright <daddywri@gmail.com>
>>> >>>
>>> >>> Hi Shigeki,
>>> >>>
>>> >>> The socket timeout exception is only a warning.  It means that some
>>> >>> site you are crawling did not accept a socket connection within
the
>>> >>> allowed time (5 minutes I think).  The Web Connector will retry
the
>>> >>> connection a few times, and if it is still rejected, it will
>>> >>> eventually give up on that page.  One thing you want to check,
>>> >>> though,
>>> >>> is that you are using proper throttling, because if you aren't then
>>> >>> one cause of this problem is that the webmaster of the site you
are
>>> >>> trying to crawl may have blocked you from accessing it.
>>> >>>
>>> >>> The database exception is more problematic.  It means that MySQL
>>> >>> thinks it took too long for a specific transaction to complete,
and
>>> >>> the database aborted the transaction due to a timeout.  There are
two
>>> >>> ways of dealing with this issue.  One way is to modify your MySQL
>>> >>> configuration to increase the transaction timeout value to some
high
>>> >>> number.  The second way is to modify ManifoldCF to recognize the
>>> >>> timeout error specifically, and cause a retry.  But in order to
do
>>> >>> the
>>> >>> latter, I would need to know what SQL error code MySQL returns for
>>> >>> this situation, which will mean we either need to look it up (if
we
>>> >>> can), or modify a ManifoldCF instance to log it when this problem
>>> >>> occurs.
>>> >>>
>>> >>> Please let me know how you would like to proceed.
>>> >>>
>>> >>> Karl
>>> >>>
>>> >>> On Wed, Oct 10, 2012 at 3:51 AM, Shigeki Kobayashi
>>> >>> <shigeki.kobayashi3@g.softbank.co.jp> wrote:
>>> >>> >
>>> >>> > Hi
>>> >>> >
>>> >>> > I am having a trouble with crawling web using MCF1.0.
>>> >>> > I run MCF with MySQL 5.5 and Tomcat 6.0.
>>> >>> > It should keep crawling contents, but MCF prints the following
>>> >>> > Database
>>> >>> > exception log, then hangs.
>>> >>> > After DB Exception, Socket Time Exception occurs.
>>> >>> >
>>> >>> > Anyone has faced this problem?
>>> >>> >
>>> >>> > --Database Exception log:
>>> >>> >
>>> >>> > ERROR 2012-10-10 16:11:05,787 (Worker thread '42') - Worker
thread
>>> >>> > aborting
>>> >>> > and restarting due to database connection reset: Database
>>> >>> > exception:
>>> >>> > Exception doing query: Lock wait timeout exceeded; try restarting
>>> >>> > transaction
>>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException:
Database
>>> >>> > exception: Exception doing query: Lock wait timeout exceeded;
try
>>> >>> > restarting
>>> >>> > transaction
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessAcivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
>>> >>> > Caused by: java.sql.SQLException: Lock wait timeout exceeded;
try
>>> >>> > restarting
>>> >>> > transaction
>>> >>> >         at
>>> >>> > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
>>> >>> >         at
>>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
>>> >>> >         at
>>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
>>> >>> >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
>>> >>> >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
>>> >>> >         at
>>> >>> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
>>> >>> >         at
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>>> >>> > ERROR 2012-10-10 16:11:06,799 (Worker thread '9') - Worker
thread
>>> >>> > aborting
>>> >>> > and restarting due to database connection reset: Database
>>> >>> > exception:
>>> >>> > Exception doing query: Lock wait timeout exceeded; try restarting
>>> >>> > transaction
>>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException:
Database
>>> >>> > exception: Exception doing query: Lock wait timeout exceeded;
try
>>> >>> > restarting
>>> >>> > transaction
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:852)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1863)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:554)
>>> >>> > Caused by: java.sql.SQLException: Lock wait timeout exceeded;
try
>>> >>> > restarting
>>> >>> > transaction
>>> >>> >         at
>>> >>> > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
>>> >>> >         at
>>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
>>> >>> >         at
>>> >>> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
>>> >>> >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
>>> >>> >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
>>> >>> >         at
>>> >>> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
>>> >>> >         at
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > ---- Socket Timeout:
>>> >>> >
>>> >>> >
>>> >>> > DEBUG 2012-10-10 16:16:27,256 (Worker thread '49') - Socket
timeout
>>> >>> > exception trying to close connection: Read timed out
>>> >>> > java.net.SocketTimeoutException: Read timed out
>>> >>> >         at java.net.SocketInputStream.socketRead0(Native Method)
>>> >>> >         at
>>> >>> > java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> >>> >         at
>>> >>> > java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> >>> >         at
>>> >>> > java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>> >>> >         at
>>> >>> > java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> >>> >         at
>>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.commons.httpclient.ChunkedInputStream.exhaustInputStream(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> >
>>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.close(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> > java.io.FilterInputStream.close(FilterInputStream.java:155)
>>> >>> >         at
>>> >>> >
>>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.close(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.close(ThrottledFetcher.java:2082)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:176)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:745)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>>> >>> >  INFO 2012-10-10 16:16:27,273 (Worker thread '49') - WEB: FETCH
>>> >>> >
>>> >>> >
>>> >>> > URL|http://xxxxxx/...|1349852786744+600514|-104|4125|org.apache.manifoldcf.core.interfaces.ManifoldCFException|
>>> >>> > Interrupted: Socket timeout: Read timed out
>>> >>> > DEBUG 2012-10-10 16:16:27,273 (Worker thread '49') - WEB: Fetch
>>> >>> > exception
>>> >>> > for 'http://xxxxxx/...'
>>> >>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>>> >>> > Interrupted:
>>> >>> > Socket timeout: Read timed out
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.noteInterrupted(ThrottledFetcher.java:1818)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:797)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>>> >>> > Caused by:
>>> >>> > org.apache.manifoldcf.agents.interfaces.ServiceInterruption:
>>> >>> > Socket timeout: Read timed out
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:101)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:745)
>>> >>> >         ... 1 more
>>> >>> > Caused by: java.net.SocketTimeoutException: Read timed out
>>> >>> >         at java.net.SocketInputStream.socketRead0(Native Method)
>>> >>> >         at
>>> >>> > java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> >>> >         at
>>> >>> > java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>>> >>> >         at
>>> >>> > java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> >>> >         at
>>> >>> > org.apache.commons.httpclient.ContentLengthInputStream.read(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> > java.io.FilterInputStream.read(FilterInputStream.java:116)
>>> >>> >         at
>>> >>> > org.apache.commons.httpclient.AutoCloseInputStream.read(Unknown
>>> >>> > Source)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2012)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:1976)
>>> >>> >         at
>>> >>> >
>>> >>> >
>>> >>> > org.apache.manifoldcf.crawler.connectors.webcrawler.DataCache.addData(DataCache.java:95)
>>> >>> >         ... 2 more
>>> >>> >  WARN 2012-10-10 16:16:27,274 (Worker thread '49') - Pre-ingest
>>> >>> > service
>>> >>> > interruption reported for job 1349774325961 connection 'WEB':
>>> >>> > Socket
>>> >>> > timeout: Read timed out
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > Regards,
>>> >>> >
>>> >>> > Shigeki
>>> >>
>>> >>
>>> >>
>>> >>
>>
>>
>>
>>
>
>
>

Mime
View raw message