lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Estrada <estrada.a...@gmail.com>
Subject Re: Dataimport performance
Date Wed, 15 Dec 2010 14:25:54 GMT
What version of Solr are you using?

Adam

2010/12/15 Robert Gr√ľndler <robert@dubture.com>

> Hi,
>
> we're looking for some comparison-benchmarks for importing large tables
> from a mysql database (full import).
>
> Currently, a full-import of ~ 8 Million rows from a MySQL database takes
> around 3 hours, on a QuadCore Machine with 16 GB of
> ram and a Raid 10 storage setup. Solr is running on a apache tomcat
> instance, where it is the only app. The tomcat instance
> has the following memory-related java_opts:
>
> -Xms4096M -Xmx5120M
>
>
> The data-config.xml looks like this (only 1 entity):
>
>      <entity name="track" query="select t.id as id, t.title as title,
> l.title as label from track t left join label l on (l.id = t.label_id)
> where t.deleted = 0" transformer="TemplateTransformer">
>        <field column="title" name="title_t" />
>        <field column="label" name="label_t" />
>        <field column="id" name="sf_meta_id" />
>        <field column="metaclass" template="Track" name="sf_meta_class"/>
>        <field column="metaid" template="${track.id}" name="sf_meta_id"/>
>        <field column="uniqueid" template="Track_${track.id}"
> name="sf_unique_id"/>
>
>        <entity name="artists" query="select a.name as artist from artist a
> left join track_artist ta on (ta.artist_id = a.id) where ta.track_id=${
> track.id}">
>          <field column="artist" name="artists_t" />
>        </entity>
>
>      </entity>
>
>
> We have the feeling that 3 hours for this import is quite long - regarding
> the performance of the server running solr/mysql.
>
> Are we wrong with that assumption, or do people experience similar import
> times with this amount of data to be imported?
>
>
> thanks!
>
>
> -robert
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message