lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Smith <sh...@thena.net>
Subject Re: Read Time Out Exception while trying to upload a huge SOLR input xml
Date Thu, 01 Apr 2010 22:18:43 GMT
The error might be that your http client doesn't handle really large
files (32-bit overflow in the Content-Length header?) or something in
your network is killing your long-lived socket?  Solr can definitely
accept a 13GB xml document.

I've uploaded large files into Solr successfully, including recently a
12GB XML input file with ~4 million documents.  My Solr instance had
2GB of memory and it took about 2 hours.  Solr streamed the XML in
nicely.  I had to jump through a couple of hoops, but in my case it
was easier than writing a tool to split up my 12GB XML file...

1. I tried to use curl to do the upload, but it didn't handle files
that large.  For my quick and dirty testing, netcat (nc) did the
trick--it doesn't buffer the file in memory and it doesn't overflow
the Content-Length header.  Plus I could pipe the data through pv to
get a progress bar and estimated time of completion.  Not recommended
for production!

  FILE=documents.xml
  SIZE=$(stat --format %s $FILE)
  (echo "POST /solr/update HTTP/1.1
  Host: localhost:8983
  Content-Type: text/xml
  Content-Length: $SIZE
  " ; cat $FILE ) | pv -s $SIZE | nc localhost 8983

2. Indexing seemed to use less memory if I configured Solr to auto
commit periodically in solrconfig.xml.  This is what I used:

    <updateHandler class="solr.DirectUpdateHandler2">
        <autoCommit>
            <maxDocs>25000</maxDocs> <!-- maximum uncommited docs
before autocommit triggered -->
            <maxTime>300000</maxTime> <!-- 5 minutes, maximum time (in
MS) after adding a doc before an autocommit is triggered -->
        </autoCommit>
    </updateHandler>

Shawn

On Thu, Apr 1, 2010 at 10:10 AM, Erick Erickson <erickerickson@gmail.com> wrote:
> Don't do that. For many reasons <G>. By trying to batch so many docs
> together, you're just *asking* for trouble. Quite apart from whether it'll
> work once, having *any* HTTP-based protocol work reliably with 13G is
> fragile...
>
> For instance, I don't want to have my know whether the XML parsing in
> SOLR parses the entire document into memory before processing or
> not. But I sure don't want my application to change behavior if SOLR
> changes it's mind and wants to process the other way. My perfectly
> working application (assuming an event-driven parser) could
> suddenly start requiring over 13G of memory... Oh my aching head!
>
> Your specific error might even be dependent upon GCing, which will
> cause it to break differently, sometimes, maybe......
>
> So do break things up and transmit multiple documents. It'll save you
> a world of hurt.
>
> HTH
> Erick
>
> On Thu, Apr 1, 2010 at 4:34 AM, Mark Fletcher
> <mark.fletcher2004@gmail.com>wrote:
>
>> Hi,
>>
>> For the first time I tried uploading a huge input SOLR xml having about 1.2
>> million *docs* (13GB in size). After some time I get the following
>> exception:-
>>
>>  <u>The server encountered an internal error ([was class
>> java.net.SocketTimeoutException] Read timed out
>> java.lang.RuntimeException: [was class java.net.SocketTimeoutException]
>> Read
>> timed out
>>  at
>>
>> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
>>  at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
>>  at
>>
>> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
>>  at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
>>  at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:279)
>>  at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138)
>>  at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>  at
>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>>  at
>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>  at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>>  at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>>  at
>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>  at
>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>  at
>>
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>  at
>>
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>  at
>>
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>  at
>>
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>  at
>>
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>  at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>  at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
>>  at
>>
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>>  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>>  at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.SocketTimeoutException: Read timed out
>> ...
>>
>> Was the file I tried to upload too big and should I try reducing its
>> size..?
>>
>> Thanks and Rgds,
>> Mark.
>>
>

Mime
View raw message