Return-Path: 
 <solr-user-return-15654-apmail-lucene-solr-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-solr-user-archive@locus.apache.org
Received: (qmail 29661 invoked from network); 11 Nov 2008 23:23:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 11 Nov 2008 23:23:45 -0000
Received: (qmail 29086 invoked by uid 500); 11 Nov 2008 23:23:48 -0000
Delivered-To: apmail-lucene-solr-user-archive@lucene.apache.org
Received: (qmail 29054 invoked by uid 500); 11 Nov 2008 23:23:48 -0000
Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:solr-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:solr-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:solr-user@lucene.apache.org>
List-Id: <solr-user.lucene.apache.org>
Reply-To: solr-user@lucene.apache.org
Delivered-To: mailing list solr-user@lucene.apache.org
Received: (qmail 29041 invoked by uid 99); 11 Nov 2008 23:23:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Nov 2008 15:23:48 -0800
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of giriprakash@gmail.com
 designates 74.125.46.29 as permitted sender)
Received: from [74.125.46.29] (HELO yw-out-2324.google.com) (74.125.46.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Nov 2008 23:22:29 +0000
Received: by yw-out-2324.google.com with SMTP id 3so74645ywj.5
        for <solr-user@lucene.apache.org>;
 Tue, 11 Nov 2008 15:23:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:mime-version:content-type;
        bh=tRJQ2R+KLu4goHuxEZN6Ul3Bfwj5sXtTCzeijUnjxTM=;
        b=m38sshnM26NLRC1mHadKUS7MhAyEmcAO/iYlRRe2o3L3q2xbIE51etxBTk/2K2jQij
         HH9Ect1w9qUNJesIUEC2B6JcAahbyolK0KeyLuxXnNSP+2A68a8yeuTKDr/M/nW+TRmY
         wWdl7dOosDJhvMYIX0QaX8HZsADav154ml9ts=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=K0TvJ+m1RUnCcEhqutqEK5TnrSNoCKjiiVMUvvzNnmVe5g3lUWSZFaCG8N9mkBX5RW
         NFBZlpnjkcC3/+J4OQcas9dVIRBvLD+nr1jZQo+Ce/hEZFgkyfAQzdKGvXECBV6TEOZM
         2c/8v9JRt5epcoOftD0F9QAjuq51gvxykveUA=
Received: by 10.90.69.7 with SMTP id r7mr7727604aga.102.1226445792168;
        Tue, 11 Nov 2008 15:23:12 -0800 (PST)
Received: by 10.90.87.11 with HTTP; Tue, 11 Nov 2008 15:23:12 -0800 (PST)
Message-ID: <844f7d420811111523s1e6b25d3ld645f921abf8e786@mail.gmail.com>
Date: Tue, 11 Nov 2008 18:23:12 -0500
From: Giri <giriprakash@gmail.com>
To: solr-user@lucene.apache.org
Subject: DataImportHandler not indexing all the records
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_136672_26251788.1226445792145"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_136672_26251788.1226445792145
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,

I have about ~ 2 million records in a mySQL database table (about 9 fields
from a single table), and I am trying to load it to the solr using
DataImportHandler using the command=full-import option. it only indexed
about 615360 records out of 2 millions.

here is my db-data-config.xml
<dataConfig>
    <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw" batchSize
="-1"/>
    <document name="climate">
        <entity name="occurence" query="select * from mylargetable">
            <field column="id" name="id" />
            <field column="title" name="title" />
            <field column="url" name="url" />
         </entity>
    </document>
</dataConfig>

and in my solr schema.xml, i define these fields as:

    <field name="id" type="string" indexed="true" stored="true"
multiValued="true"/>
    <field name="title" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
    <field name="url" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>


If I try to index just one field (id), then it indexes about 960000 records,
but if I try to index all the above three fields, it indexes only 615360
records.

Any help will be appreciated.

thanks!

------=_Part_136672_26251788.1226445792145--