Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of
 SRS0=WZmi+o=GO=basetechnology.com=jack@yourhostingaccount.com designates
 65.254.253.77 as permitted sender)
Message-ID: <CF87B2B5AD5F4ECB864CCAAB482177B3@JackKrupansky>
From: "Jack Krupansky" <jack@basetechnology.com>
To: <java-user@lucene.apache.org>
References: <20120814114536.209050@gmx.net>
In-Reply-To: <20120814114536.209050@gmx.net>
Subject: Re: Solr adding Documents / Commit in different Threads
Date: Tue, 14 Aug 2012 08:11:07 -0400
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="UTF-8";
	reply-type=original
Content-Transfer-Encoding: 7bit
Importance: Normal
Sender: "Jack Krupansky" <jack@basetechnology.com>

Try sending only 100 or so (or maybe even only 20) of the documents at a 
time, and only send commit with the last batch.

Sometimes network-related components along the way have trouble dealing with 
very large requests unless carefully configured. In other words, maybe you 
can find a way to make such a large request work, but it may not be worth 
the effort.

-- Jack Krupansky

-----Original Message----- 
From: Ralf Heyde
Sent: Tuesday, August 14, 2012 7:45 AM
To: java-user@lucene.apache.org
Subject: Solr adding Documents / Commit in different Threads

Hello,

we currently facing a problem which may lost updates for some documents 
during adding / comitting.

The infrastructure: we have a main solr, which gets documents and distribute 
them to a lot of slaves.

The situation:  we have a Job, which runs scheduled every minute (no run, if 
a previous job is not ended), looks for new / modified documents and adds, 
deletes and finally commits them to the Main Solr Server. They are added due 
SolrServer.add(Collection<SolrInputDocument>). This Collection contains up 
to 10.000 documents - so the request could take some time.

Additionally we have a second job, which only runs commits every minute.

Today i found a stacktrace in our logs. Can someone tell me, whether this is 
a problem? Is it possible that these two processes conflict each other (e.g. 
lost document updates).

Regards,

Ralf

---------- Stracktrace -----------

2012-08-14 10:05:50,141  WARN ... 
org.apache.solr.handler.ReplicationHandler$FileStream.write(ReplicationHandler.java:1066): 
Exception while writing response for params: 
checksum=true&wt=filestream&indexversion=1330453083759&file=_eah.frq&command=filecontent
ClientAbortException:  java.io.IOException
        at 
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:373)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:356)
        at 
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
        at 
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
        at 
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:93)
        at 
org.apache.solr.common.util.FastOutputStream.write(FastOutputStream.java:87)
        at 
org.apache.solr.handler.ReplicationHandler$FileStream.write(ReplicationHandler.java:1054)
        at 
org.apache.solr.handler.ReplicationHandler$3.write(ReplicationHandler.java:914)
        at 
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at 
de.zalando.solr.shared.experimentalfeature.ExperimentalFeatureFilter.doFilter(ExperimentalFeatureFilter.java:28)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)
        at 
org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:257)
        at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
        at 
org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1764)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException
        at 
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:208)
        at 
org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:238)
        at 
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119)
        at 
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:190)
        at org.apache.coyote.Response.doWrite(Response.java:533)
        at 
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
        ... 27 more


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org