lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mannott, Birgit" <B.Mann...@klopotek.de>
Subject RE: [bulk]: Re: Dates and DataImportHandler
Date Thu, 21 Sep 2017 11:23:39 GMT
Forget it. I mixed up the timezone with the locale. I thought I could set the timezone for
the DIH last index but that's not possible

-----Original Message-----
From: Mannott, Birgit [mailto:B.Mannott@klopotek.de] 
Sent: Thursday, September 21, 2017 1:09 PM
To: solr-user@lucene.apache.org
Subject: [bulk]: [bulk]: Re: Dates and DataImportHandler

As far as I understood, you can use the locale so that DIH saves the last index time for 
the given time zone and not for UTC. So if you set the locale according to the timezone of
your DB you don't need to convert dates for comparison.

But for me it's not working because every time I include settings for the propertyWriter (concrete
data doesn't matter - I tried a lot),  the null pointer exception appears. So I had to find
a workaround and convert now the time saved by the DIH from UTC to the timezone of my DB when
comparing for delta imports. But it's an ugly workaround I don't like. 

So, if anyone has an idea why this NPE occurs it would be great. Do I perhaps have to add
something to solrconfig.xml?

Thanks,
Birgit



-----Original Message-----
From: Jamie Jackson [mailto:jamiejaxon@gmail.com]
Sent: Tuesday, September 19, 2017 6:54 PM
To: solr-user@lucene.apache.org
Subject: [bulk]: Re: [bulk]: Dates and DataImportHandler

FWIW, I know mine worked, so maybe try:

<propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss"
type="SimplePropertiesWriter" />

I can't conceive of what the locale would possibly do when a dateFormat is specified, so I
omitted the attribute. (Maybe one can specify dateFormat *or *locale--it seems like specifying
both would cause a clash.) For what it's worth, the format you're trying to write seems identical
to the default*, so I'm not sure what benefit you're getting by using that propertyWriter.

*It's identical to *my* default, anyway. Maybe the default changes based on one's system configuration,
I don't know. This stuff isn't very well documented.

On Tue, Sep 19, 2017 at 7:22 AM, Mannott, Birgit <B.Mannott@klopotek.de>
wrote:

> Hi,
>
> I have a similar problem. I try to change the timezone for the 
> last_index_time by setting
>
>         <propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss"
> type="SimplePropertiesWriter" locale="en_US" />
>
> in the <dataConfig> section of my data-config.xml file.
>
> But when doing this I always get a NullPointerException on Delta Import:
>
> 2017-09-15 14:04:00.825 INFO  (Thread-2938) [   x:mex_prd_dev1100-ap]
> o.a.s.h.d.DataImporter Starting Delta Import
> 2017-09-15 14:04:00.827 ERROR (Thread-2938) [   x:mex_prd_dev1100-ap]
> o.a.s.h.d.DataImporter Delta Import Failed
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable 
> to PropertyWriter implementation:SimplePropertiesWriter
>             at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:330)
>             at org.apache.solr.handler.dataimport.DataImporter.
> doDeltaImport(DataImporter.java:439)
>             at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:476)
>             at org.apache.solr.handler.dataimport.DataImporter.
> lambda$runAsync$0(DataImporter.java:457)
>             at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>             at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:
> 598)
>             at org.apache.solr.handler.dataimport.
> SimplePropertiesWriter.init(SimplePropertiesWriter.java:100)
>             at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:328)
>             ... 4 more
>
> Has anyone an idea what is wrong or missing?
>
> Thanks,
> Birgit
>
>
>
> -----Original Message-----
> From: Jamie Jackson [mailto:jamiejaxon@gmail.com]
> Sent: Tuesday, September 19, 2017 3:42 AM
> To: solr-user@lucene.apache.org
> Subject: [bulk]: Dates and DataImportHandler
>
> Hi folks,
>
> My DB server is on America/Chicago time. Solr (on Docker) is running 
> on UTC. Dates coming from my (MariaDB) data source seem to get 
> translated properly into the Solr index without me doing anything special.
>
> However when doing delta imports using last_index_time ( 
> http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport
> ), I can't seem to get the date, which Solr provides, to be understood 
> by the DB as being UTC (and translated back, accordingly). In other 
> words, the DB thinks the Solr UTC date is local, so it thinks the date 
> is ahead by six hours.
>
> '${dataimporter.request.clean}' != 'false'
>
> or dt > '${dataimporter.last_index_time}'
>
> I came up with this workaround, which seems to work:
>
> '${dataimporter.request.clean}' != 'false'
>
> /* ${user.timezone} is UTC, and the
> ${custom.dataimporter.datasource.tz}
> property is set to America/Chicago */
>
> or dt > CONVERT_TZ('${dataimporter.last_index_time}','${user.
> timezone}','${
> custom.dataimporter.datasource.tz}')
>
> However, isn't there a way for this translation to happen more naturally?
>
> I thought maybe I could do something like this:
>
> <propertyWriter
>
> dateFormat="yyyy-MM-dd HH:mm:ssZ"
>
> type="SimplePropertiesWriter"
>
> />
>
> The above did set the property as expected (with a trailiing `+0000`), 
> but that didn't seem to help the DB understand/translate the date.
>
> Thanks,
> Jamie
>
Mime
View raw message