Return-Path: Delivered-To: apmail-lucene-solr-user-archive@minotaur.apache.org Received: (qmail 39287 invoked from network); 6 Nov 2009 02:02:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 6 Nov 2009 02:02:32 -0000 Received: (qmail 64243 invoked by uid 500); 6 Nov 2009 02:02:30 -0000 Delivered-To: apmail-lucene-solr-user-archive@lucene.apache.org Received: (qmail 64163 invoked by uid 500); 6 Nov 2009 02:02:30 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 64151 invoked by uid 99); 6 Nov 2009 02:02:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 02:02:30 +0000 X-ASF-Spam-Status: No, hits=-2.5 required=5.0 tests=AWL,BAYES_00,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yseeley@gmail.com designates 209.85.219.226 as permitted sender) Received: from [209.85.219.226] (HELO mail-ew0-f226.google.com) (209.85.219.226) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 02:02:27 +0000 Received: by ewy26 with SMTP id 26so776074ewy.5 for ; Thu, 05 Nov 2009 18:02:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:content-type:content-transfer-encoding; bh=WJ1xllZAzZp1Zow5AlWUxdBT9FtHDgxJ8BFC3KH+46A=; b=l4snHrPv2thW4/4ZCynwOoQXfXGTm8KLNDI4dH7o2J+2RGcIyFarvu6k7SXpTXiyQ9 JXlF38TjMIHRoHwJjW14/prC2aKa9pHiPFgqkS3ratEmdrOhfptP4iuYwS4jgk+VfsDy ufVpz3qx0fgKGedicJnFen4O4uNHFAbGiFqZU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=w8p6Ce7GxHas2Ci8GcyJtOEEkLIsgziXs06g04acn30XQ/0/IpQfIIjFjGAlpJHujU wHr3lpeAazfGwRJOGbvOcbj5L0zeBBHKI9wcezYKWcc+khBUZTjqrVuvtHY5RCg3D6Sz QoEgMLnxTL/7vsCI5S7QJiUUk0ci4Xgil0lCY= MIME-Version: 1.0 Sender: yseeley@gmail.com Reply-To: yonik@lucidimagination.com Received: by 10.216.87.134 with SMTP id y6mr1304235wee.20.1257472926043; Thu, 05 Nov 2009 18:02:06 -0800 (PST) In-Reply-To: <69de18140911020305i4163f0c2m660e15eb1b0f7319@mail.gmail.com> References: <628F7059F6E90442BD2D9CA1601D354004CDBD89@ENTWEXMB0000000.university.harvard.edu> <87c998320910131759j7aaf8081n19646b2c2dba7ea7@mail.gmail.com> <628F7059F6E90442BD2D9CA1601D354004F53A58@ENTWEXMB0000000.university.harvard.edu> <69de18140911020305i4163f0c2m660e15eb1b0f7319@mail.gmail.com> Date: Thu, 5 Nov 2009 21:02:05 -0500 X-Google-Sender-Auth: 20767398028dba7a Message-ID: Subject: Re: StreamingUpdateSolrServer - indexing process stops in a couple of hours From: Yonik Seeley To: solr-user@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Seems fixed. https://issues.apache.org/jira/browse/SOLR-1543 -Yonik http://www.lucidimagination.com On Mon, Nov 2, 2009 at 6:05 AM, Shalin Shekhar Mangar wrote: > I'm able to reproduce this issue consistently using JDK 1.6.0_16 > > After an optimize is called, only one thread keeps adding documents and t= he > rest wait on StreamingUpdateSolrServer line 196. > > On Sun, Oct 25, 2009 at 8:03 AM, Dadasheva, Olga > wrote: > >> I am using java 1.6.0_05 >> >> To illustrate what is happening I wrote this test program that has 10 >> threads adding a collection of documents and one thread optimizing the i= ndex >> every 10 sec. >> >> I am seeing that after the first optimize there is only one thread that >> keeps adding documents. The other ones are locked. >> >> In the real code I ended up adding synchronized around add on optimize t= o >> avoid this. >> >> public static void main(String[] args) { >> >> =A0 =A0 =A0 =A0final JettySolrRunner jetty =3D new JettySolrRunner("/sol= r", 8983 ); >> =A0 =A0 =A0 =A0try { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0jetty.start(); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// setup the server... >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0String url =3D "http://localhost:8983/sol= r"; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0final StreamingUpdateSolrServer server = =3D new >> StreamingUpdateSolrServer( url, 2, 5 ) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 @Override >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public void handleError(T= hrowable ex) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 // do so= methign... >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0server.setConnectionTimeout(1000); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0server.setDefaultMaxConnectionsPerHost(10= 0); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0server.setMaxTotalConnections(100); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0int i =3D 0; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0while (i++ < 10) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0new Thread("add-thread"+i= ) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public vo= id run(){ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0int j =3D 0; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0while (true) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0try { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0List docs >> =3D new ArrayList(); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0for (int n =3D 0; n < 50; n++) >> { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0SolrInputDocument doc =3D >> new SolrInputDocument(); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0String docID =3D >> this.getName()+"_doc_"+j++; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0doc.addField( "id", >> docID); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0doc.addField( "content", >> "document_"+docID); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0docs.add(doc); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0server.add(docs); >> >> =A0System.out.println(this.getName()+" added "+docs.size()+" documents")= ; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0Thread.sleep(100); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0} catch (Exception e) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); >> >> =A0System.err.println(this.getName()+" "+e.getLocalizedMessage()); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0System.exit(0); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}.start(); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0new Thread("optimizer-thread") { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public void run(){ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0while (tr= ue) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0Thread.sleep(10000); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0server.optimize(); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0System.out.println(this.getName()+" >> optimized"); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch (= Exception e) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0e.printStackTrace(); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0System.err.println("optimizer >> "+e.getLocalizedMessage()); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0System.exit(0); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}.start(); >> >> >> =A0 =A0 =A0 =A0} catch (Exception e) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); >> =A0 =A0 =A0 =A0 } >> >> } >> -----Original Message----- >> From: Lance Norskog [mailto:goksron@gmail.com] >> Sent: Tuesday, October 13, 2009 8:59 PM >> To: solr-user@lucene.apache.org >> Subject: Re: StreamingUpdateSolrServer - indexing process stops in a cou= ple >> of hours >> >> Which Java release is this? =A0There are known thread-blocking problems = in >> Java 1.5. >> >> Also, what sockets are used during this time? Try 'netstat -s | fgrep 89= 83' >> (or your Solr URL port #) and watch the active, TIME_WAIT, CLOSE_WAIT >> sockets build up. This may give a hint. >> >> On Tue, Oct 13, 2009 at 8:47 AM, Dadasheva, Olga < >> olga_dadasheva@harvard.edu> wrote: >> > Hi, >> > >> > I am indexing documents using StreamingUpdateSolrServer. My 'setup' >> > code is almost a copy of the junit test of the Solr trunk. >> > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0try { >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0StreamingUpdateSolrServ= er streamingServer =3D new >> > StreamingUpdateSolrServer( url, 2, 5 ) { >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0@Override >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0public void handleError= (Throwable ex) { >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.= out.println(" new >> > StreamingUpdateSolrServer error "+ex); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mail.se= nd(new >> > Date()+"StreamingUpdateSolrServer error. "+ex); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}; >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0streamingServer.setConnecti= onTimeout(20*60*1000); >> > // 20 min >> > >> > streamingServer.setDefaultMaxConnectionsPerHost(100); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0streamingServer.setMaxTotal= Connections(100); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0server =3D streamingServer; >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} catch(Exception e) { >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mail.send(new Date()+"StreamingUpdateSo= lrServer error. >> > "+e); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0e.printStackTrace(); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> > >> > >> > This is what happens: >> > >> > The crawler is running for a couple+/- of hours and I am seeing that >> > the thread number is being reused , but in general it keeps increasing >> > up to >> > thread-119 >> > >> > >> > 2009-10-09 16:53:24,532 INFO =A0starting runner: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@def >> > 14 f (StreamingUpdateSolrServer.java:86) - [pool-1-thread-1] >> > 2009-10-09 16:53:24,543 INFO =A0starting runner: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@34b >> > 35 0 (StreamingUpdateSolrServer.java:86) - [pool-1-thread-2] >> > 2009-10-09 16:53:25,338 INFO =A0finished: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@34b >> > 35 0 (StreamingUpdateSolrServer.java:164) - [pool-1-thread-2] >> > 2009-10-09 16:53:25,504 INFO =A0finished: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@def >> > 14 f (StreamingUpdateSolrServer.java:164) - [pool-1-thread-1] .... >> > 2009-10-09 18:22:49,844 INFO =A0finished: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@563 >> > d5 >> > 6 (StreamingUpdateSolrServer.java:164) - [pool-1-thread-117] >> > 2009-10-09 18:22:49,848 INFO =A0finished: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@10f >> > ff 00 (StreamingUpdateSolrServer.java:164) - [pool-1-thread-119] >> > 2009-10-09 18:22:49,859 INFO =A0finished: >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@10e >> > 52 >> > c1 (StreamingUpdateSolrServer.java:164) - [pool-1-thread-118] >> > >> > At this point the log stops. >> > >> > >> > >> > Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode): >> > >> > "btpool0-70" prio=3D3 tid=3D0x0078a800 nid=3D0xd7 in Object.wait() >> > [0x8a5ff000..0x8a5ff970] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool. >> > ja >> > va:482) >> > =A0 =A0 =A0 =A0- locked <0xb0a7d620> (a >> > org.mortbay.thread.BoundedThreadPool$PoolThread) >> > >> > "Java2D Disposer" daemon prio=3D3 tid=3D0x0033f400 nid=3D0x21 in >> > Object.wait() [0x893ff000..0x893ff870] >> > =A0 java.lang.Thread.State: WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.j= ava:116) >> > =A0 =A0 =A0 =A0- locked <0xabc000f8> (a java.lang.ref.ReferenceQueue$L= ock) >> > =A0 =A0 =A0 =A0at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.j= ava:132) >> > =A0 =A0 =A0 =A0at sun.java2d.Disposer.run(Disposer.java:125) >> > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619) >> > >> > "MultiThreadedHttpConnectionManager cleanup" daemon prio=3D3 >> > tid=3D0x002afc00 nid=3D0x1f in Object.wait() [0x897ff000..0x897ff970] >> > =A0 java.lang.Thread.State: WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0- waiting on <0xabc001c8> (a java.lang.ref.ReferenceQue= ue$Lock) >> > =A0 =A0 =A0 =A0at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.j= ava:116) >> > =A0 =A0 =A0 =A0- locked <0xabc001c8> (a java.lang.ref.ReferenceQueue$L= ock) >> > =A0 =A0 =A0 =A0at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.j= ava:132) >> > =A0 =A0 =A0 =A0at >> > org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$Refer >> > en >> > ceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122) >> > >> > "DefaultQuartzScheduler_QuartzSchedulerThread" prio=3D3 tid=3D0x00cb5c= 00 >> > nid=3D0x1c in Object.wait() [0x899ff000..0x899ffbf0] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:4 >> > 33 >> > ) >> > =A0 =A0 =A0 =A0- locked <0xabc695d8> (a java.lang.Object) >> > >> > "DefaultQuartzScheduler_Worker-10" prio=3D3 tid=3D0x00e26c00 nid=3D0x1= b in >> > Object.wait() [0x89aff000..0x89affb70] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc83ac0> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-9" prio=3D3 tid=3D0x00dd8400 nid=3D0x1a= in >> > Object.wait() [0x89bff000..0x89bff8f0] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc83920> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-8" prio=3D3 tid=3D0x0074ac00 nid=3D0x19= in >> > Object.wait() [0x89cff000..0x89cff870] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc83780> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-7" prio=3D3 tid=3D0x0074a400 nid=3D0x18= in >> > Object.wait() [0x89dff000..0x89dff9f0] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc835e0> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-6" prio=3D3 tid=3D0x00826800 nid=3D0x17= in >> > Object.wait() [0x89eff000..0x89eff970] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc83440> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-5" prio=3D3 tid=3D0x0074dc00 nid=3D0x16= in >> > Object.wait() [0x89fff000..0x89fffaf0] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc832a0> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-4" prio=3D3 tid=3D0x00e04400 nid=3D0x15= in >> > Object.wait() [0x8a0ff000..0x8a0ffa70] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc83100> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-3" prio=3D3 tid=3D0x006b4800 nid=3D0x14= in >> > Object.wait() [0x8a1ff000..0x8a1ffbf0] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc82f60> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-2" prio=3D3 tid=3D0x00879800 nid=3D0x13= in >> > Object.wait() [0x8a2ff000..0x8a2ffb70] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :519) >> > =A0 =A0 =A0 =A0- locked <0xabc82dc0> (a >> > org.quartz.simpl.SimpleThreadPool$WorkerThread) >> > >> > "DefaultQuartzScheduler_Worker-1" prio=3D3 tid=3D0x00e51c00 nid=3D0x12 >> > waiting on condition [0x8a3ff000..0x8a3ff8f0] >> > =A0 java.lang.Thread.State: WAITING (parking) >> > =A0 =A0 =A0 =A0at sun.misc.Unsafe.park(Native Method) >> > =A0 =A0 =A0 =A0- parking to wait for =A0<0xac07acb8> (a >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. >> > aw >> > ait(AbstractQueuedSynchronizer.java:1925) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java: >> > 25 >> > 4) >> > =A0 =A0 =A0 =A0at >> > org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(St >> > re >> > amingUpdateSolrServer.java:199) >> > =A0 =A0 =A0 =A0at >> > org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(Abs >> > tr >> > actUpdateRequest.java:89) >> > =A0 =A0 =A0 =A0at >> > org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49) >> > =A0 =A0 =A0 =A0at >> > harvard.solr.crawl.AbstractSearchService.update(AbstractSearchService. >> > ja >> > va:180) >> > =A0 =A0 =A0 =A0at harvard.solr.crawl.Updater.internalUpdate(Updater.ja= va:97) >> > =A0 =A0 =A0 =A0at harvard.solr.crawl.Updater.update(Updater.java:34) >> > =A0 =A0 =A0 =A0at >> > harvard.solr.crawl.crawler.SiteStructureCrawler.start(SiteStructureCra >> > wl >> > er.java:71) >> > =A0 =A0 =A0 =A0at >> > harvard.solr.crawl.job.NewIndexJob.execute(NewIndexJob.java:36) >> > =A0 =A0 =A0 =A0at org.quartz.core.JobRunShell.run(JobRunShell.java:202= ) >> > =A0 =A0 =A0 =A0at >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja >> > va >> > :525) >> > >> > "pool-2-thread-1" prio=3D3 tid=3D0x00232400 nid=3D0xf waiting on condi= tion >> > [0x8a6ff000..0x8a6ff970] >> > =A0 java.lang.Thread.State: WAITING (parking) >> > =A0 =A0 =A0 =A0at sun.misc.Unsafe.park(Native Method) >> > =A0 =A0 =A0 =A0- parking to wait for =A0<0xac06ffc0> (a >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. >> > aw >> > ait(AbstractQueuedSynchronizer.java:1925) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java >> > :3 >> > 58) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav= a: >> > 946) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. >> > ja >> > va:906) >> > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619) >> > >> > "Timer-0" prio=3D3 tid=3D0x0083b800 nid=3D0xe in Object.wait() >> > [0x8a7ff000..0x8a7ffaf0] >> > =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at java.util.TimerThread.mainLoop(Timer.java:509) >> > =A0 =A0 =A0 =A0- locked <0xac073d30> (a java.util.TaskQueue) >> > =A0 =A0 =A0 =A0at java.util.TimerThread.run(Timer.java:462) >> > >> > "btpool0-0 - Acceptor0 SocketConnector @ 0.0.0.0:8983" prio=3D3 >> > tid=3D0x00826000 nid=3D0xd runnable [0x8a8ff000..0x8a8ffa70] >> > =A0 java.lang.Thread.State: RUNNABLE >> > =A0 =A0 =A0 =A0at java.net.PlainSocketImpl.socketAccept(Native Method) >> > =A0 =A0 =A0 =A0at java.net.PlainSocketImpl.accept(PlainSocketImpl.java= :384) >> > =A0 =A0 =A0 =A0- locked <0xabe279d8> (a java.net.SocksSocketImpl) >> > =A0 =A0 =A0 =A0at java.net.ServerSocket.implAccept(ServerSocket.java:4= 53) >> > =A0 =A0 =A0 =A0at java.net.ServerSocket.accept(ServerSocket.java:421) >> > =A0 =A0 =A0 =A0at >> > org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97) >> > =A0 =A0 =A0 =A0at >> > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.jav= a: >> > 516) >> > =A0 =A0 =A0 =A0at >> > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool. >> > ja >> > va:442) >> > >> > "Low Memory Detector" daemon prio=3D3 tid=3D0x00118800 nid=3D0xb runna= ble >> > [0x00000000..0x00000000] >> > =A0 java.lang.Thread.State: RUNNABLE >> > >> > "CompilerThread1" daemon prio=3D3 tid=3D0x00116000 nid=3D0xa waiting o= n >> > condition [0x00000000..0x8aefec18] >> > =A0 java.lang.Thread.State: RUNNABLE >> > >> > "CompilerThread0" daemon prio=3D3 tid=3D0x00115000 nid=3D0x9 waiting o= n >> > condition [0x00000000..0x8affeb98] >> > =A0 java.lang.Thread.State: RUNNABLE >> > >> > "Signal Dispatcher" daemon prio=3D3 tid=3D0x00113400 nid=3D0x8 waiting= on >> > condition [0x00000000..0x00000000] >> > =A0 java.lang.Thread.State: RUNNABLE >> > >> > "Finalizer" daemon prio=3D3 tid=3D0x000f9400 nid=3D0x7 in Object.wait(= ) >> > [0x8b1ff000..0x8b1ff970] >> > =A0 java.lang.Thread.State: WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.j= ava:116) >> > =A0 =A0 =A0 =A0- locked <0xac2e46c0> (a java.lang.ref.ReferenceQueue$L= ock) >> > =A0 =A0 =A0 =A0at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.j= ava:132) >> > =A0 =A0 =A0 =A0at >> > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >> > >> > "Reference Handler" daemon prio=3D3 tid=3D0x000f5000 nid=3D0x6 in >> > Object.wait() [0x8b2ff000..0x8b2ffaf0] >> > =A0 java.lang.Thread.State: WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Object.java:485) >> > =A0 =A0 =A0 =A0at >> > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >> > =A0 =A0 =A0 =A0- locked <0xabc87150> (a java.lang.ref.Reference$Lock) >> > >> > "main" prio=3D3 tid=3D0x00030000 nid=3D0x2 in Object.wait() >> > [0xfe67f000..0xfe67fd80] >> > =A0 java.lang.Thread.State: WAITING (on object monitor) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> > =A0 =A0 =A0 =A0- waiting on <0xac07c6f0> (a java.lang.Object) >> > =A0 =A0 =A0 =A0at java.lang.Object.wait(Object.java:485) >> > =A0 =A0 =A0 =A0at harvard.solr.crawl.Scheduler.schedule(Scheduler.java= :61) >> > =A0 =A0 =A0 =A0- locked <0xac07c6f0> (a java.lang.Object) >> > =A0 =A0 =A0 =A0at harvard.solr.crawl.Scheduler.main(Scheduler.java:99) >> > >> > "VM Thread" prio=3D3 tid=3D0x000f1400 nid=3D0x5 runnable >> > >> > "GC task thread#0 (ParallelGC)" prio=3D3 tid=3D0x00037000 nid=3D0x3 ru= nnable >> > >> > "GC task thread#1 (ParallelGC)" prio=3D3 tid=3D0x00038800 nid=3D0x4 ru= nnable >> > >> > "VM Periodic Task Thread" prio=3D3 tid=3D0x0011a400 nid=3D0xc waiting = on >> > condition >> > >> > JNI global references: 689 >> > >> > Heap >> > =A0PSYoungGen =A0 =A0 =A0total 417792K, used 27487K [0xe1400000, 0xfc0= 00000, >> > 0xfc000000) >> > =A0eden space 403456K, 6% used [0xe1400000,0xe2becae8,0xf9e00000) >> > =A0from space 14336K, 20% used [0xfb200000,0xfb4eb208,0xfc000000) >> > =A0to =A0 space 17408K, 0% used [0xf9e00000,0xf9e00000,0xfaf00000) >> > =A0PSOldGen =A0 =A0 =A0 =A0total 876544K, used 86178K [0xabc00000, 0xe= 1400000, >> > 0xe1400000) >> > =A0object space 876544K, 9% used [0xabc00000,0xb1028b90,0xe1400000) >> > =A0PSPermGen =A0 =A0 =A0 total 131072K, used 44021K [0x8bc00000, 0x93c= 00000, >> > 0xabc00000) >> > =A0object space 131072K, 33% used [0x8bc00000,0x8e6fd460,0x93c00000) >> > >> >> >> >> -- >> Lance Norskog >> goksron@gmail.com >> > > > > -- > Regards, > Shalin Shekhar Mangar. >