Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of john.wang@gmail.com
 designates 72.14.220.153 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=NrC3xZOhoEBNzP29y4bxRpcmpR2Pjw2kvIFoD7g+OKtmEfRhHsMh3CswjM8WTSkdKlp/Nd8GDsYQyeNWL/vvGpOC8+tYpqug1pCaUOJk0+/YyCIYkHiNo9eg80TBO+/NvtOTNFmvWNMELd/rH3pahqrU1OtLfARPTGWg/7A1dkE=
Message-ID: <8837fb770804030736w235452fcwde3f684d5722d105@mail.gmail.com>
Date: Thu, 3 Apr 2008 07:36:42 -0700
From: "John Wang" <john.wang@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: payload performance wrt fieldcache
In-Reply-To: <8837fb770804030727w474f9a7epb4131bfc29716fa6@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_1490_31027156.1207233402134"
References: <8837fb770804030727w474f9a7epb4131bfc29716fa6@mail.gmail.com>

------=_Part_1490_31027156.1207233402134
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Sorry, gmail was screwy and accidentally sent the msg.
Anyway,

I have a large index, about 30M docs.
I have a date field (by days) and there are about 1000 of them, every doc
has a date field filled in.

So out of curiosity I index the date field two ways:
1) using "date" as a field, and set the date value for each doc.
2) new term: "_payload:_val" and added the date (as a long or 8 byte array)
into the payload of each doc.

loading into an array long[] of length maxdoc of dates, the performance was
surprising:
using payload is 7 times slower than using fieldcache.

At first I thought it was because of the conversion between byte[8] to a
long for each doc, I changed it so it loads into byte[8*maxdoc] without
doing the conversion, and the result is the same.

I then did another experiment:
lower the number of dates down to a small number, e.g. 50, and timed field
cache load, and it took much longer than when it had 1000.

I did some profiling and the profiler is pointing to TermPositions.next
and TermPositions.nextPosition and TermPositions.getPayload as the culprit.

I would think payload would always be faster. Any ideas?

Thanks
-John

On Thu, Apr 3, 2008 at 7:27 AM, John Wang <john.wang@gmail.com> wrote:

> Hi:
>
>

------=_Part_1490_31027156.1207233402134--