lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giri <giriprak...@gmail.com>
Subject DataImportHandler not indexing all the records
Date Tue, 11 Nov 2008 23:23:12 GMT
Hi,

I have about ~ 2 million records in a mySQL database table (about 9 fields
from a single table), and I am trying to load it to the solr using
DataImportHandler using the command=full-import option. it only indexed
about 615360 records out of 2 millions.

here is my db-data-config.xml
<dataConfig>
    <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw" batchSize
="-1"/>
    <document name="climate">
        <entity name="occurence" query="select * from mylargetable">
            <field column="id" name="id" />
            <field column="title" name="title" />
            <field column="url" name="url" />
         </entity>
    </document>
</dataConfig>

and in my solr schema.xml, i define these fields as:

    <field name="id" type="string" indexed="true" stored="true"
multiValued="true"/>
    <field name="title" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
    <field name="url" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>


If I try to index just one field (id), then it indexes about 960000 records,
but if I try to index all the above three fields, it indexes only 615360
records.

Any help will be appreciated.

thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message