lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Matheis <ste...@mathe.is>
Subject Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters
Date Tue, 15 Aug 2017 15:46:42 GMT
Birgit,

any chance to utilise one of the caching strategies that DIH offers?

Like building a complete map for one of the subentities? That would mean
reading the whole table at the beginning and then only doing lookups by key.

Or getting data from subentities with joins in your main entity?

Heavily depends on the amount of data we're talking about - but might be
worth a thought.

Best
-Stefan

On Aug 15, 2017 4:33 PM, "Erick Erickson" <erickerickson@gmail.com> wrote:

I presume you're using Data Import Handler? An alternative when you
get into complex imports is to use a SolrJ client, here's a sample.
That way you can use whatever tools the particular JDBC connector will
allow and can be much faster.

https://lucidworks.com/2012/02/14/indexing-with-solrj/

Best,
Erick

On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit <B.Mannott@klopotek.de>
wrote:
> Hi,
>
> I'm using solr 6.6.0 and I have to do a complex data import from an
oracle db concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52
Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by
creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.
>
> I think, the main remaining problem is that oracle cursor sharing is not
used and that for every select a hard parse is done.
> Solr does not use binding variables. This would be the easiest way to use
oracle cursor sharing. But I didn't find anything about influencing the way
solr builds select statements.
> I could force oracle cursor sharing without binding variables but I have
to do this configuration for the session. I'm not allowed to change the
configuration of the whole database system.
>
> Is there a way to execute a command like "ALTER SESSION SET
cursor_sharing = FORCE;" after getting the connection for processing an
entity?
>
> Thanks,
> Birgit

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message