Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of trejkaz@trypticon.org
 designates 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAL8PwkaBd0miN-vKcCPpYHt+fQfuDxs74At6R4vkSXXUts1y9A@mail.gmail.com>
References: 
 <CAPYoNwerH3wnqz7YAGZ5ixKQ8NY7agPZEA_DB4t49uvJGr6g2A@mail.gmail.com>
	<CAL8Pwkb32KbGbuz5XvzwZkJrNOoyKZw7iAOW3d6AO3JYuHwBmQ@mail.gmail.com>
	<CAPYoNwesZM2pZfY=iOb8tB7vfkO2xr-xG7CXUd55NpTsPV3Akw@mail.gmail.com>
	<CAL8PwkaBd0miN-vKcCPpYHt+fQfuDxs74At6R4vkSXXUts1y9A@mail.gmail.com>
Date: Tue, 14 Oct 2014 16:29:32 +1100
Message-ID: 
 <CADuAvzf0Kxdb=Uctqe89s8vd5HRkEsXW7WUyQhVd-PG729=5+A@mail.gmail.com>
Subject: Re: ArrayIndexOutOfBoundsException: -65536
From: Trejkaz <trejkaz@trypticon.org>
To: Lucene Users Mailing List <java-user@lucene.apache.org>
Content-Type: text/plain; charset=UTF-8

Bit of thread necromancy here, but I figured it was relevant because
we get exactly the same error.

On Thu, Jan 19, 2012 at 12:47 AM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Hmm, are you certain your RAM buffer is 3 MB?
>
> Is it possible you are indexing an absurdly enormous document...?

We're seeing a case here where the document certainly could qualify as
"absurdly enormous". The doc itself is 2GB in size and the
tokenisation is per-character, not per-word, so the number of
generated terms must be enormous. Probably enough to fill 2GB...

So I'm wondering if there is more info somewhere on why this is (or
was? We're still using 3.6.x) a limit and whether it can be detected
up-front. Some large amount of indexing time (~30 minutes) could be
avoided if we can detect that it would have failed ahead of time.

TX

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org