lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yangrui Guo <guoyang...@gmail.com>
Subject Re: data import extremely slow
Date Sat, 07 Nov 2015 17:02:18 GMT
I just realized that not everything was ok. Three child entities were not
imported. Had set batchSize to -1 but again solr was stuck :(

On Fri, Nov 6, 2015 at 3:11 PM, Yangrui Guo <guoyangrui@gmail.com> wrote:

> Thanks for the reply. I just removed CacheKeyLookUp and CachedKey and used
> WHERE clause instead. Everything works fine now.
>
> Yangrui
>
>
> On Friday, November 6, 2015, Shawn Heisey <apache@elyograg.org> wrote:
>
>> On 11/6/2015 10:32 AM, Yangrui Guo wrote:
>> > <entity name="movie_actress" transformer="RegexTransformer"
>>
>> There's a good chance that JDBC is trying to read the entire result set
>> (all three million rows) into memory before sending any of that info to
>> Solr.
>>
>> Set the batchSize to -1 for MySQL so that it will stream results to Solr
>> as soon as they are available, and not wait for all of them.  Here's
>> more info on the situation, which frequently causes OutOfMemory problems
>> for users:
>>
>>
>> http://wiki.apache.org/solr/DataImportHandlerFaq?highlight=%28mysql%29|%28batchsize%29#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F
>> <http://wiki.apache.org/solr/DataImportHandlerFaq?highlight=%28mysql%29%7C%28batchsize%29#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F>
>>
>>
>> Thanks,
>> Shawn
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message