lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Blythe <j...@curvolabs.com>
Subject import efficiencies
Date Thu, 26 May 2016 16:01:00 GMT
hi all,

i've got layered entities in my solr import. it's calling on some
transactional data from a MySQL instance. there are two fields that are
used to then lookup other information from other tables via their related
UIDs, one of which has its own child entity w yet another select statement
to grab up more data.

it fetches at about 120/s but processes at ~50-60/s. we currently only have
close to 500k records, but it's growing quickly and thus is becoming
increasingly painful to make modifications due to the reimport that needs
to then occur.

i feel like i'd seen some threads regarding commits of new data,
master/slave, or solrcloud/sharding that could help in some ways related to
this but as of yet can't scrounge them up w my searches (ironic :p).

can someone help by pointing me to some good material related to this sort
of thing?

thanks-

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message