lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: Indexing into SolrCloud
Date Thu, 18 Jul 2013 16:49:04 GMT
Sorry, but you need to resend this message to the Solr user list - this is 
the Lucene user list.

-- Jack Krupansky

-----Original Message----- 
From: Beale, Jim (US-KOP)
Sent: Thursday, July 18, 2013 12:34 PM
To: java-user@lucene.apache.org
Subject: Indexing into SolrCloud

Hey folks,

I've been migrating an application which indexes about 15M documents from 
straight-up Lucene into SolrCloud.  We've set up 5 Solr instances with a 3 
zookeeper ensemble using HAProxy for load balancing. The documents are 
processed on a quad core machine with 6 threads and indexed into SolrCloud 
through HAProxy using ConcurrentUpdateSolrServer in order to batch the 
updates.  The indexing box is heavily-loaded

I've been accepting the default HttpClient with 50K buffered docs and 2 
threads, i.e.,

int solrMaxBufferedDocs = 50000;
int solrThreadCount = 2;
solrServer = new ConcurrentUpdateSolrServer(solrHttpIPAddress, 
solrMaxBufferedDocs, solrThreadCount);

autoCommit is configured in the solrconfig as follows:

     <autoCommit>
       <maxTime>600000</maxTime>
       <maxDocs>500000</maxDocs>
       <openSearcher>false</openSearcher>
     </autoCommit>

I'm getting the following errors on the client and server sides 
respectively:

Client side:

2013-07-16 19:02:47,002 [concurrentUpdateScheduler-1-thread-4] INFO 
SystemDefaultHttpClient - I/O exception (java.net.SocketException) caught 
when processing request: Software caused connection abort: socket write 
error
2013-07-16 19:02:47,002 [concurrentUpdateScheduler-1-thread-4] INFO 
SystemDefaultHttpClient - Retrying request
2013-07-16 19:02:47,002 [concurrentUpdateScheduler-1-thread-5] INFO 
SystemDefaultHttpClient - I/O exception (java.net.SocketException) caught 
when processing request: Software caused connection abort: socket write 
error
2013-07-16 19:02:47,002 [concurrentUpdateScheduler-1-thread-5] INFO 
SystemDefaultHttpClient - Retrying request

Server side:

7988753 [qtp1956653918-23] ERROR org.apache.solr.core.SolrCore  รข 
java.lang.RuntimeException: [was class org.eclipse.jetty.io.EofException] 
early EOF
        at 
com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
        at 
com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
        at 
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
        at 
com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
        at 
org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393)

When I disabled autoCommit on the server side, I didn't see any errors there 
but I still get the issue client-side after about 2 million documents - 
which is about 45 minutes.

Has anyone seen this issue before?  I couldn't find anything useful on the 
usual places.

I suppose I could setup wireshark to see what is happening but I'm hoping 
that someone has a better suggestion.

Thanks in advance for any help!


Best regards,
Jim Beale

hibu.com
2201 Renaissance Boulevard, King of Prussia, PA, 19406
Office: 610-879-3864
Mobile: 610-220-3067

The information contained in this email message, including any attachments, 
is intended solely for use by the individual or entity named above and may 
be confidential. If the reader of this message is not the intended 
recipient, you are hereby notified that you must not read, use, disclose, 
distribute or copy any part of this communication. If you have received this 
communication in error, please immediately notify me by email and destroy 
the original message, including any attachments. Thank you.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message