lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <karl.wri...@nokia.com>
Subject RE: Solr spewage and dropped documents, while indexing
Date Mon, 07 Jun 2010 15:05:16 GMT
Perhaps - although missing_content_stream seems to imply that it had at least partly read 4
requests which later failed.  Also, wouldn't there be something in the output log which would
give us a clue as to what happened?

Is there any post-hiccup spelunking I can reasonably do?  Or should I try to reproduce the
problem with more diagnostics on?

Karl


-----Original Message-----
From: ext Bernd Fondermann [mailto:bf_jak@brainlounge.de] 
Sent: Monday, June 07, 2010 10:54 AM
To: dev@lucene.apache.org
Subject: Re: Solr spewage and dropped documents, while indexing

Looks like a server-side problem to me.
Maybe the server ran out of sockets or other resources and just replied
with a 400 error?

  Bernd

karl.wright@nokia.com wrote:
> Hi folks,
> 
> This morning I was experimenting with using multiple threads while indexing some 20,000,000
records worth of content.  In fact, my test spun up some 50 threads, and happily chugged away
for a couple of hours before I saw the following output from my test code:
> 
> Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to index record
6469124
> Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to index record
6469551
> Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to index record
6470592
> Http protocol error: HTTP/1.1 400 missing_content_stream, while trying to index record
6472454
> java.net.SocketException: Connection reset
>         at java.net.SocketInputStream.read(SocketInputStream.java:168)
>         at HttpPoster.getResponse(HttpPoster.java:280)
>         at HttpPoster.indexPost(HttpPoster.java:191)
>         at ParseAndLoad$PostThread.run(ParseAndLoad.java:638)
> <<<<<<
> 
> Looking at the solr-side output, I see nothing interesting at all:
> 
> Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update/extract params={literal.nokia_longitude=9.78518981933594&literal.nokia_phone=%2B497971910474&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_district=Münster&literal.nokia_placerating=0&literal.id=6472724&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=1&literal.nokia_ppid=276u0wyw-c8cb7f4d6cd84a639a4e7d3570bf8814&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9985514322917&literal.nokia_postalcode=74405&literal.nokia_street=WeinhaldenstraÃe&literal.nokia_title=Dorfgemeinschaft+Münster+e.V.&literal.nokia_category=261}
status=0 QTime=1
> Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update/extract params={literal.nokia_longitude=9.76717020670573&literal.nokia_phone=%2B497971950725&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_placerating=0&literal.id=6472737&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=13&literal.nokia_ppid=276u0wyw-d3bed6449fcb41b0adc50ae08e041f8d&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9974405924479&literal.nokia_fax=%2B497971950712&literal.nokia_postalcode=74405&literal.nokia_street=KochstraÃe&literal.nokia_title=BayWa+AG+Bau-+%26+Gartenmarkt&literal.nokia_category=194}
status=0 QTime=0
> Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update/extract params={literal.nokia_longitude=9.77591044108073&literal.nokia_phone=%2B49797124009&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_district=Unterrot&literal.nokia_placerating=0&literal.id=6472739&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=28&literal.nokia_ppid=276u0wyw-d534d7a9235a4edf878d5e32a34bad8b&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9791788736979&literal.nokia_fax=%2B49797123431&literal.nokia_postalcode=74405&literal.nokia_street=HauptstraÃe&literal.nokia_title=Gastel+R.&literal.nokia_category=5}
status=0 QTime=1
> Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update/extract params={literal.nokia_longitude=9.76935&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_placerating=5&literal.id=6472698&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=15&literal.nokia_ppid=276u0wyw-9544100e68d74162aff54783b9376134&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9981&literal.nokia_postalcode=74405&literal.nokia_street=KanzleistraÃe&literal.nokia_tag=Steuerberater&literal.nokia_tag=Business+%26+Service&literal.nokia_title=Consultis+GmbH&literal.nokia_category=215}
status=0 QTime=92
> Jun 7, 2010 9:57:48 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update/extract params={literal.nokia_longitude=9.77173970540364&literal.nokia_phone=%2B4979713238&literal.nokia_type=0&literal.nokia_boost=1&literal.nokia_placerating=0&literal.id=6472699&literal.nokia_visitcount=0&literal.nokia_country=DEU&literal.nokia_housenumber=37&literal.nokia_ppid=276u0wyw-9600016fd0d248c9b442111838350f64&literal.nokia_language=de&literal.nokia_city=Gaildorf&literal.nokia_latitude=48.9987182617188&literal.nokia_fax=%2B497971911639&literal.nokia_postalcode=74405&literal.nokia_street=KarlstraÃe&literal.nokia_title=Videothek,+5th+avenue+Peltekis+Apostolos&literal.nokia_category=5}
status=0 QTime=93
> <<<<<<
> 
> It is unlikely (but, of course, not out of the question) that this hiccup is due to some
reentrancy problem in my test code.  It is much more likely to be some kind of a Solr multi-threaded
race condition - especially since it looks like a number of requests all failed at precisely
the same time.  This is a Solr 1.5 build from mid-late March, FWIW.  Does anyone know of an
extractingUpdateRequestHandler re-entrancy bug of this kind?
> 
> Thanks,
> Karl
> 
> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message