Mailing-List: contact user-help@lucy.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@lucy.apache.org
Received-SPF: pass (nike.apache.org: domain of dag@nimrod.no designates
 195.139.160.2 as permitted sender)
Sender: dag@nimrod.no
To: user@lucy.apache.org
References: <8CC8A0E7-1CE0-40BE-9127-7966E45162F6@gmail.com>
From: Dag Lem <dag@nimrod.no>
Organization: Nimrod
Date: 23 Oct 2012 14:35:10 +0200
In-Reply-To: <8CC8A0E7-1CE0-40BE-9127-7966E45162F6@gmail.com>
Message-ID: <8662614b9d.fsf@sid.nimrod.no>
Lines: 24
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.4
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Subject: Re: [lucy-user] Running out of memory while loading

Ken Youens-Clark <kyclark@gmail.com> writes:

> Hi,
> 
> I'm just experimenting with the use of Lucy to index the data I formally store in relational (MySQL) databases.  I'm just taking the text from the db and putting into Lucy stores.  Each directory gets its own directory so that it's easy for me to update just part of my search index when a db changes.  So far, I've processed about 30 dbs into a total of about 2.3GB of Lucy indexes.  The problem is that my machine (a pretty decent, dual-core Linux host) keeps running out of memory, esp. when indexing a large db with 100K+ records.  My sysadmin keeps killing it as the it will take down the machine.  
> 
> I'm using the latest Perl and Lucy source.  Any ideas?
> 
> Ken
> 

One possible cause of your problems is that your database driver
attempts to blast the complete result set from your SQL query into
client memory in one go, so that you are likely to run out of memory
before even starting on the indexing.

I don't really know DBD::mysql, assuming that's what you're using, but
I know from first hand experience that you have to take special care
with DBD::Pg.

-- 
Best regards,

Dag Lem