Return-Path: 
 <java-user-return-32393-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-java-user-archive@www.apache.org
Received: (qmail 49791 invoked from network); 4 Feb 2008 15:53:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 4 Feb 2008 15:53:41 -0000
Received: (qmail 71083 invoked by uid 500); 4 Feb 2008 15:53:29 -0000
Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org
Received: (qmail 70106 invoked by uid 500); 4 Feb 2008 15:53:27 -0000
Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:java-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:java-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:java-user@lucene.apache.org>
List-Id: <java-user.lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Delivered-To: mailing list java-user@lucene.apache.org
Received: (qmail 70095 invoked by uid 99); 4 Feb 2008 15:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2008 07:53:27 -0800
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of erickerickson@gmail.com
 designates 209.85.128.189 as permitted sender)
Received: from [209.85.128.189] (HELO fk-out-0910.google.com) (209.85.128.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2008 15:52:59 +0000
Received: by fk-out-0910.google.com with SMTP id z23so1984727fkz.5
        for <java-user@lucene.apache.org>;
 Mon, 04 Feb 2008 07:53:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        bh=10Mvwg/vS5VZYoP3T3UZxc3ETI1/liovUVq1te506iU=;
        b=IvumYo+OBmXwm/WrXd/qEjON4Oz1bRYCaYUh5tQse/rx7fuazJb7rCoPdz83wFa+OcBCZf6xMEyuyeRy3D4UvDvptXM0fghf2ThuSr/cQTLFkSJFFeAuZLyWgPap22fKVJ9gziBay2biBfnSMalUFP7KgZQXMCFP7gGw5JSJiWw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=PkhQGcHvzn3eDk1cLn0i+96DhHpY0IRKliEX23kc3wHfqZ+waeRmW5Pd/3KeOdb36GsXBBuVw1edn4s6aZ3/3x/lRCYPDPL50dn3xFcnhMttAYrTE1jcuaUszZP45MshimsMak0E9NewWRuuUeJEXTyXCpmqCe06xSny8Jiw7Us=
Received: by 10.82.107.15 with SMTP id f15mr13337494buc.0.1202140384999;
        Mon, 04 Feb 2008 07:53:04 -0800 (PST)
Received: by 10.82.151.7 with HTTP; Mon, 4 Feb 2008 07:53:04 -0800 (PST)
Message-ID: <359a92830802040753g6f691030pa1221ac727d8a3be@mail.gmail.com>
Date: Mon, 4 Feb 2008 10:53:04 -0500
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: outof memory error
In-Reply-To: <4628d2a90802040725v33ffd76x3638e0c34343190@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_3994_21775820.1202140384990"
References: <4628d2a90802040725v33ffd76x3638e0c34343190@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_3994_21775820.1202140384990
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

ummmm index smaller documents? <G>

You cannot expect to index a 1G doc with 512M of memory in the JVM.
The first thing I'd try is upping your JVM memory to the max your machine
will accept.

Make sure you flush your IndexWriter before attempting to index this
document.

But I would not be surprised if this failed to solve the problem. What's in
this massive document? Would it be possible to break it up into
smaller segments and index many sub-documents for this massive doc?
I also wonder what problem you're trying to solve by indexing this doc.
Is it a log file? I can't imagine a text document that big. That's like a
100 volume encyclopedia, and I can't help but wonder whether your users
would be better served by indexing it in pieces.

Best
Erick

On Feb 4, 2008 10:25 AM, SK R <rsk.sen@gmail.com> wrote:

> Hi,
>   I got outof memory exception while  indexing  huge documents (~1GB) in
> one thread and optimizing some other (2 to 3) indexes in different
> threads.
> Max JVM heap size is 512MB. I'm using lucene2.3.0.
>
>   Please suggest a way to avoid this exception.
>
> Regards
>  RSK
>

------=_Part_3994_21775820.1202140384990--