Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Message-ID;
  b=rkXrr55IHVHUt5pDblQ2X4xMm8wJ9ZnTMZMeC91V0NYgt6DlD6HSrwRl4Tody2bM7/hSlZzp6LuN37yGtxVPBPdhkOF+ZwajnJ5Cn8p59a+jWwNyWb9nw9cTr0wUeAs8pd8EOONsfoIrWJVrKHxDcbspJdmHN7TXlNEtshmCD+Q=;
Date: Fri, 12 Dec 2008 14:15:57 -0800 (PST)
From: Kay Kay <kaykay.unique@yahoo.com>
Reply-To: kaykay.unique@yahoo.com
Subject: Re: Solr - DataImportHandler - Large Dataset results ?
To: solr-user@lucene.apache.org
In-Reply-To: <69de18140812121341t27a77cbejab6d95d999573eaf@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-2038308922-1229120157=:24606"
Message-ID: <151701.24606.qm@web111216.mail.gq1.yahoo.com>

--0-2038308922-1229120157=:24606
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

I am using MySQL. I believe (since MySQL 5) supports streaming.=20

On more about streaming - can we assume that when the database driver suppo=
rts streaming , the resultset iterator is a forward directional iterator.=
=20

If , say the streaming size is 10K records and we are trying to retrieve a =
total of 100K records - what exactly happens when the threshold is reached =
, (say , the first 10K records were retrieved ).=20

Are the previous set of records thrown away and replaced in memory by the n=
ew batch of records.=A0=20


--- On Fri, 12/12/08, Shalin Shekhar Mangar <shalinmangar@gmail.com> wrote:
From: Shalin Shekhar Mangar <shalinmangar@gmail.com>
Subject: Re: Solr - DataImportHandler - Large Dataset results ?
To: solr-user@lucene.apache.org
Date: Friday, December 12, 2008, 9:41 PM

DataImportHandler is designed to stream rows one by one to create Solr
documents. As long as your database driver supports streaming, you should b=
e
fine. Which database are you using?

On Sat, Dec 13, 2008 at 2:20 AM, Kay Kay <kaykay.unique@yahoo.com> wrote:

> As per the example in the wiki -
> http://wiki.apache.org/solr/DataImportHandler  - I am seeing the followin=
g
> fragment.
>
> <dataSource driver=3D"org.hsqldb.jdbcDriver"
> url=3D"jdbc:hsqldb:/temp/example/ex" user=3D"sa" />
>    <document name=3D"products">
>        <entity name=3D"item" query=3D"select * from
item">
>            <field column=3D"ID" name=3D"id" />
>            <field column=3D"NAME" name=3D"name" />
>              ......................
>    </entity>
> </document>
> </dataSource>
>
> My scaled-down application looks very similar along these lines but where
> my resultset is so big that it cannot fit within main memory by any
chance.
>
> So I was planning to split this single query into multiple subqueries -
> with another conditional based on the id . ( id < 0 and id > 100 ,
say ) .
>
> I am curious if there is any way to specify another conditional clause ,
> (<splitData Column =3D "id"  batch=3D"10000" />,
where the column is supposed to
> be an integer value) - and internally , the implementation could actually
> generate the subqueries -
>
> i) get the min , max of the numeric column , and send queries to the
> database based on the batch size
>
> ii) Add Documents for each batch and close the resultset .
>
> This might end up putting more load on the database (but at least the
> dataset would fit in the main memory ).
>
> Let me know if anyone else had run into similar issues and how this was
> encountered.
>
>
>


--=20
Regards,
Shalin Shekhar Mangar.
=0A=0A=0A      
--0-2038308922-1229120157=:24606--