lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vivek sar <vivex...@gmail.com>
Subject Re: httpclient.ProtocolException using Solrj
Date Thu, 09 Apr 2009 17:06:08 GMT
I'm inserting 10K in a batch (using addBeans method). I read somewhere
in the wiki that it's better to use the same instance of SolrServer
for better performance. Would MultiThreadedConnectionManager help? How
do I use it?

I also wanted to know how can use EmbeddedSolrServer - does my app
needs to be running in the same jvm with Solr webapp?

Thanks,
-vivek

2009/4/9 Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@gmail.com>:
> how many documents are you inserting ?
> may be you can create multiple instances of CommonshttpSolrServer and
> upload in parallel
>
>
> On Thu, Apr 9, 2009 at 11:58 AM, vivek sar <vivextra@gmail.com> wrote:
>> Thanks Shalin and Paul.
>>
>> I'm not using MultipartRequest. I do share the same SolrServer between
>> two threads. I'm not using MultiThreadedHttpConnectionManager. I'm
>> simply using CommonsHttpSolrServer to create the SolrServer. I've also
>> tried StreamingUpdateSolrServer, which works much faster, but does
>> throws "connection reset" exception once in a while.
>>
>> Do I need to use MultiThreadedHttpConnectionManager? I couldn't find
>> anything on it on Wiki.
>>
>> I was also thinking of using EmbeddedSolrServer - in what case would I
>> be able to use it? Does my application and the Solr web app need to
>> run into the same JVM for this to work? How would I use the
>> EmbeddedSolrServer?
>>
>> Thanks,
>> -vivek
>>
>>
>> On Wed, Apr 8, 2009 at 10:46 PM, Shalin Shekhar Mangar
>> <shalinmangar@gmail.com> wrote:
>>> Vivek, do you share the same SolrServer instance between your two threads?
>>> If so, are you using the MultiThreadedHttpConnectionManager when creating
>>> the HttpClient instance?
>>>
>>> On Wed, Apr 8, 2009 at 10:13 PM, vivek sar <vivextra@gmail.com> wrote:
>>>
>>>> single thread everything works fine. Two threads are fine too for a
>>>> while and all the sudden problem starts happening.
>>>>
>>>> I tried indexing using REST services as well (instead of Solrj), but
>>>> with that too I get following error after a while,
>>>>
>>>> 2009-04-08 10:04:08,126 ERROR [indexerThreadPool-5] Indexer -
>>>> indexData()-> Failed to index
>>>> java.net.SocketException: Broken pipe
>>>>        at java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>        at
>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>>>>        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>>>>        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>        at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
>>>>        at
>>>> org.apache.commons.httpclient.methods.StringRequestEntity.writeRequest(StringRequestEntity.java:145)
>>>>        at
>>>> org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
>>>>         at
>>>> org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
>>>>        at
>>>> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
>>>>        at
>>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
>>>>        at
>>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>>>>        at
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>>>>        at
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
>>>>
>>>>
>>>> Note, I'm using "simple" lock type. I'd tried "single" type before
>>>> that once caused index corruption so I switched to "simple".
>>>>
>>>> Thanks,
>>>> -vivek
>>>>
>>>> 2009/4/8 Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@gmail.com>:
>>>> > do you see the same problem when you use a single thread?
>>>> >
>>>> > what is the version of SolrJ that you use?
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Apr 8, 2009 at 1:19 PM, vivek sar <vivextra@gmail.com>
wrote:
>>>> >> Hi,
>>>> >>
>>>> >>  Any ideas on this issue? I ran into this again - once it starts
>>>> >> happening it keeps happening. One of the thread keeps failing. Here
>>>> >> are my SolrServer settings,
>>>> >>
>>>> >>        int socketTO = 0;
>>>> >>        int connectionTO = 100;
>>>> >>        int maxConnectionPerHost = 10;
>>>> >>        int maxTotalConnection = 50;
>>>> >>        boolean followRedirects = false;
>>>> >>        boolean allowCompression = true;
>>>> >>        int maxRetries = 1;
>>>> >>
>>>> >> Note, I'm using two threads to simultaneously write to the same
index.
>>>> >>
>>>> >> org.apache.solr.client.solrj.SolrServerException:
>>>> >> org.apache.commons.httpclient.ProtocolException: Unbuffered entity
>>>> >> enclosing request can not be repeated.
>>>> >>        at
>>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:470)
>>>> >>        at
>>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242)
>>>> >>        at
>>>> org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259)
>>>> >>        at
>>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48)
>>>> >>        at
>>>> org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:57)
>>>> >>
>>>> >> Thanks,
>>>> >> -vivek
>>>> >>
>>>> >> On Sat, Apr 4, 2009 at 1:07 AM, vivek sar <vivextra@gmail.com>
wrote:
>>>> >>> Hi,
>>>> >>>
>>>> >>>  I'm sending 15K records at once using Solrj (server.addBeans(...))
>>>> >>> and have two threads writing to same index. One thread goes
fine, but
>>>> >>> the second thread always fails with,
>>>> >>>
>>>> >>>
>>>> >>> org.apache.solr.client.solrj.SolrServerException:
>>>> >>> org.apache.commons.httpclient.ProtocolException: Unbuffered
entity
>>>> >>> enclosing request can not be repeated.
>>>> >>>        at
>>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:470)
>>>> >>>        at
>>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242)
>>>> >>>        at
>>>> org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259)
>>>> >>>        at
>>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48)
>>>> >>>        at
>>>> org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:57)
>>>> >>>        at
>>>> com.apple.afterchat.indexer.solr.handler.BeanIndexHandler.indexData(BeanIndexHandler.java:44)
>>>> >>>        at
>>>> com.apple.afterchat.indexer.Indexer.indexData(Indexer.java:77)
>>>> >>>        at com.apple.afterchat.indexer.Indexer.run(Indexer.java:39)
>>>> >>>        at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>> >>>        at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>> >>>        at java.lang.Thread.run(Thread.java:637)
>>>> >>> Caused by: org.apache.commons.httpclient.ProtocolException:
Unbuffered
>>>> >>> entity enclosing request can not be repeated.
>>>> >>>        at
>>>> org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:487)
>>>> >>>        at
>>>> org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
>>>> >>>        at
>>>> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
>>>> >>>        at
>>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
>>>> >>>        at
>>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>>>> >>>        at
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>>>> >>>        at
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
>>>> >>>        at
>>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:417)
>>>> >>>
>>>> >>> Does anyone know what could be the problem?
>>>> >>>
>>>> >>> Thanks,
>>>> >>> -vivek
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > --Noble Paul
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>
>
>
> --
> --Noble Paul
>

Mime
View raw message