lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dyer, James" <James.D...@ingrambook.com>
Subject RE: DIH - unable to ADD individual new documents
Date Tue, 03 Jul 2012 19:12:17 GMT
A DIH request handler can only process one "run" at a time.  So if DIH is still in process
and you kick off a new DIH "full-import" it will silently ignore the new command.  To have
more than one DIH "run" going at a time it is necessary to configure more than one handler
instance in sorlconfig.xml.  But even then you'll have to be careful to find one that is free
before trying to use it.

Regardless, to do what you want, you'll need to poll the DIH response screen to be sure it
isn't running before starting a new one.  It would be simplest to leave it with just 1 DIH
handler in solrconfig.xml.  If you've got to have an undefined # of concurrent updates going
at once you're best off to not use DIH.

Perhaps a better usage pattern for which DIH was designed for is to put the doc id's in an
update table with a timestamp.  Have your queries join to the update table "where timestamp
> ${dih.last_index_time}".  Set up crontab or whatever to kick off DIH every so often.
 If the prior run is still in progress, it will just skip that run, but because we're dealing
with timestamps that get written automatically when DIH finishes, you will only experience
a delayed update, not a lost update.  By batching your updates like this you will also have
fewer commits, which will be beneficial for performance all around.

Of course if you're trying to do this with the near-real-time functionality batching isn't
your answer.  But DIH isn't designed at all to work well with NRT either...

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-----Original Message-----
From: Klostermeyer, Michael [mailto:mklostermeyer@riskexchange.com] 
Sent: Tuesday, July 03, 2012 1:55 PM
To: solr-user@lucene.apache.org
Subject: RE: DIH - unable to ADD individual new documents

Some interesting findings over the last hours, that may change the context of this discussion...

Due to the nature of the application, I need the ability to fire off individual "ADDs" on
several different entities at basically the same time.  So, I am making 2-4 Solr ADD calls
within 100ms of each other.  While troubleshooting this, I found that if I only made 1 Solr
ADD call (ignoring the other entities), it updated the index as expected.  However, when all
were fired off, proper indexing did not occur (at least on one of the entities) and no errors
were logged.  I am still attempting to figure out if ALL of the 2-4 entities failed to ADD,
or if some failed and others succeeded.

So...does this have something to do with Solr's index/message queuing (v3.5)?  How does Solr
handle these types of rapid requests, and even more important, how do I get the status of
an individual DIH call vs simply the status of the "latest" call at /dataimport?

Mike


-----Original Message-----
From: Gora Mohanty [mailto:gora@mimirtech.com] 
Sent: Monday, July 02, 2012 10:02 PM
To: solr-user@lucene.apache.org
Subject: Re: DIH - unable to ADD individual new documents

On 3 July 2012 07:54, Klostermeyer, Michael <mklostermeyer@riskexchange.com> wrote:
> I should add that I am using the full-import command in all cases, and setting clean=false
for the individual adds.

What does the data-import page report at the end of the full-import, i.e., how many documents
were indexed?
Are there any error messages in the Solr logs? Please share with us your DIH configuration
file, and Solr schema.xml.

Regards,
Gora

Mime
View raw message