lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gian Maria Ricci - aka Alkampfer <alkamp...@nablasoft.com>
Subject Best practice for incremental Data Import Handler
Date Mon, 14 Dec 2015 17:29:34 GMT
Hi,

 

I just want some feedback on best practice to run incremental DIH. During
last years I always preferred to have dedicated application that pushes data
inside ElasticSearch / Solr, but now I have a situation where we are forced
to use DIH.

 

I have several SQL Server database with a column of type timestamp (I'm
trying to understand if it is possible to have a standard DateTime column).

 

In the past I've written a super simple C# routine that executes these macro
steps

 

1)      Query solr to understand if the DIH is running (to avoid problem if
multiple instances fired togheter)

2)      Query solr to get the document with higher timestamp value

3)      Launch DIH passing the higer timestamp value to do incremental
population (Greater than or equal)

4)      Monitor DIH and wait for it to finish.

 

I never had problem with this approach, but actually I'm wondering if there
is some better approach instead of having a custom routine that manage
running DIH. Also I'm in a situation where we are not allowed to run C#
code, so we should rewrite that simple program in Node.js or plain bash
shell. My aim is not reimplementing the wheel J.

 

Thanks for any suggestion you can give me.

--
Gian Maria Ricci
Cell: +39 320 0136949

 <http://mvp.microsoft.com/en-us/mvp/Gian%20Maria%20Ricci-4025635>
<http://www.linkedin.com/in/gianmariaricci>
<https://twitter.com/alkampfer>   <http://feeds.feedburner.com/AlkampferEng>


 


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message