lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: indexing error - 6.3.0
Date Thu, 19 Jan 2017 15:28:55 GMT
It looks to me like you're using "field guessing". For production systems I
generally don't recommend this. The problem is that it makes the best estimate
that it can based on the first document for any given field. So it sees a field
with the value 1 and tries to make the field an int. Then 100 docs later a doc
comes through with that field as 1.0 and you get an indexing exception.

Next, if you're sending many docs rapidly through SolrCloud, there are all kinds
of things going on to try to update the configset, reload the cores to
get the latest
configurations down to all of the replicas and the like.

So the very first thing I'd try is to define the schema manually and see if that
cures things.

BTW, the big, scary "DO NOT EDIT THIS FILE" in the managed_schema file
is a bit of overkill. You _can_ edit that file manually, the danger is
that if you
have the field-guessing turned on, already running solr nodes may overwrite
your changes. So it's safe to manually edit that file and push it to Zookeeper
in to situations:
1> you have disabled "field guessing"
or
2> you edit and push when all your Solr nodes are shut down.

Best,
Erick

On Wed, Jan 18, 2017 at 9:11 PM, Joe Obernberger
<joseph.obernberger@gmail.com> wrote:
> Hi All - I've been trying to debug this, but it keeps occurring. Even if I
> do 100 at a time, or 50 at a time, eventually I get the below stack trace.
> I've also adjusted the autoSoftCommit and autoCommit times to a variety of
> values.  It stills fails after a time; typically around 27-50 million
> records, I get this error. This is on a newly created collection (that I've
> been dropping and recreating after each test).
>
> Is there anything I can try that may help debug?  Perhaps my method of
> indexing is incorrect?  Thanks for any ideas!
>
> -Joe
>
>
> On 1/17/2017 10:13 AM, Joe Obernberger wrote:
>>
>> While indexing a large number of records in Solr Cloud 6.3.0 with a 5 node
>> configuration, I received an error.  I'm using java code / solrj to perform
>> the indexing by creating a list of SolrInputDocuments, 1000 at a time, and
>> then calling CloudSolrClient.add(list).  The records are small - about 6
>> fields of short strings and numbers.
>>
>> If I do 100 at a time, I can't replicate the error, but 1000 at a time has
>> consistently causes the below exception to occur.  The index is stored in a
>> shared HDFS.
>>
>> 2017-01-17 04:21:00.022 ERROR (qtp606548741-21) [c:Worldline s:shard5
>> r:core_node1 x:Worldline_shard5_replica1] o.a.s.h.RequestHandlerBase
>> org.apache.solr.common.SolrException: Exception writing document id
>> 6228601a-8756-4b16-bdc3-ad026754b225 to the index; possible analysis error.
>>         at
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:178)
>>         at
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
>>         at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:957)
>>         at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1112)
>>         at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:738)
>>         at
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>>         at
>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)
>>         at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
>>         at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
>>         at
>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:275)
>>         at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
>>         at
>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:240)
>>         at
>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:158)
>>         at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
>>         at
>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
>>         at
>> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
>>         at
>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>>         at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>         at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2213)
>>         at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>>         at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>         at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>         at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>         at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>>         at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>>         at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>>         at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>>         at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>>         at
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>>         at
>> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>>         at
>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>>         at
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>>         at
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>>         at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>>         at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>         at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>> IndexWriter is closed
>>         at
>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:740)
>>         at
>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:754)
>>         at
>> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1558)
>>         at
>> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:279)
>>         at
>> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
>>         at
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
>>         ... 62 more
>> Caused by: org.apache.lucene.index.CorruptIndexException: invalid state:
>> base=49, docID=258046 (resource=_8u.cfs [slice=_8u.fdt])
>>         at
>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:559)
>>         at
>> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:200)
>>         at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:89)
>>         at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4312)
>>         at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3889)
>>         at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
>>         at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
>>
>> -Joe
>>
>

Mime
View raw message