lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Harris <>
Subject DataImportHandler Robustness For Imports That Take A Long Time
Date Mon, 09 Mar 2009 20:24:33 GMT
I have a dataset (7M-ish docs each of which is maybe 1-100K) that,
with my current indexing process, takes a few days or maybe a week to
put into Solr.  I'm considering maybe switching to indexing with the
DataImportHandler, but I'm concerned about the impact of this on
indexing robustness:

If I understand DIH properly, then if Solr goes down for whatever
reason during an import, then DIH loses track of what it has and
hasn't yet indexed that round, and will thus probably do a lot of
redundant reimporting the next time you run an import command. (For
example, if DIH successfully imports row id 100, and then Solr dies
before the DIH import finishes, and then I restart Solr and start a
new delta-import, then I think DIH will import row id 100 again.) One
implication for my dataset seems to be that, unless Solr can actually
stay up for several days on end, then DIH will never finish importing
my data, even if I manage to keep Solr at, say, 99% uptime. This would
be fine if a full import took only a few hours. If full import could
take a week, though, this is slightly unnerving. (Sometimes you just
need to restart Solr. Or the machine itself, for that matter.)

Are there any good ways around this with DIH? One potential option is
to give each row in the database table not only a
ModificationTimestamp column but also a DataImportHandlerTimestamp
column, and try to get DIH to update that column whenever it finishes
indexing a row. Then you'd modify the WHERE clause in the DIH config
so that instead of determining which rows to index with something like

  WHERE ModificationTimestamp > dataimporter.last_index_time

you'd use something like

  WHERE ModificationTimestamp > SolrImportTimestamp

In this way, hopefully, DIH can always pick up where it left off last time,
rather than trying to redo any work it might have actually managed
to do last round.

(I'm using something along these lines with my current, non-DIH-based
indexing scheme.)

Am I making sense here?


View raw message