Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of andreas.guther@gmail.com
 designates 209.85.132.248 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=tlUIc1/ACFFspwXG2mcwMlOUldcNR2jZjXrQHlY3Td36NE2NMSGwGVa8C9mhg7vzvDiNDAEpQojwOHJ78+XOaNuag3s+iIOAtuhuisKti3OfqodSZ3U+0rlhXTopzysv+zt2y26f2swDEVT2LDp1W3IL2kUilbvbxiBO28oTP/4=
Message-ID: <c38ea6f20705170643q75ebbd18if1f9882664c82be0@mail.gmail.com>
Date: Thu, 17 May 2007 06:43:30 -0700
From: "Andreas Guther" <andreas.guther@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Field.Store.Compress - does it improve performance of document
 reads?
In-Reply-To: <9FABF4D2-AB41-4D10-98D3-E5C02005B85E@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_42068_23504604.1179409410340"
References: <c38ea6f20705162310r70f5dc84ga10e781483b38cac@mail.gmail.com>
	 <200705171001.45912.paul.elschot@xs4all.nl>
	 <9FABF4D2-AB41-4D10-98D3-E5C02005B85E@apache.org>

------=_Part_42068_23504604.1179409410340
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I am actually using the FieldSelector and unless I did something wrong it
did not provide me any load performance improvements which was surprising to
me and disappointing at the same time.  The only difference I could see was
when I returned for all fields a NO_LOAD which from my understanding is the
same as skipping over the document.

Right now I am looking into fragmentation problems of my huge index files.
I am de-fragmenting the hard drive to see if this brings any read
performance improvements.

I am also wondering if the FieldCache as discussed in
http://www.gossamer-threads.com/lists/lucene/general/28252 would help
improve the situation.

Andreas

On 5/17/07, Grant Ingersoll <gsingers@apache.org> wrote:
>
> I haven't tried compression either.  I know there was some talk a
> while ago about deprecating, but that hasn't happened.  The current
> implementation yields the highest level of compression.  You might
> find better results by compressing in your application and storing as
> a binary field, thus giving you more control over CPU used.  This is
> our current recommendation for dealing w/ compression.
>
> If you are not actually displaying that field, you should look into
> the FieldSelector API (via IndexReader).  It allows you to lazily
> load fields or skip them all together and can yield a pretty
> significant savings when it comes to loading documents.
> FieldSelector is available in 2.1.
>
> -Grant
>
> On May 17, 2007, at 4:01 AM, Paul Elschot wrote:
>
> > On Thursday 17 May 2007 08:10, Andreas Guther wrote:
> >> I am currently exploring how to solve performance problems I
> >> encounter with
> >> Lucene document reads.
> >>
> >> We have amongst other fields one field (default) storing all
> >> searchable
> >> fields.  This field can become of considerable size since we are
> >> indexing
> >> documents and  store the content for display within results.
> >>
> >> I noticed that the read can be very expensive.  I wonder now if it
> >> would
> >> make sense to add this field as Field.Store.Compress to the
> >> index.  Can
> >> someone tell me if this would speed up the document read or if
> >> this is
> >> something only interesting for saving space.
> >
> > I have not tried the compression yet, but in my experience a good way
> > to reduce the costs of document reads from a disk is by reading them
> > in document number order whenever possible. In this way one saves
> > on the disk head seeks.
> > Compression should actually help reducing the costs of disk head seeks
> > even more.
> >
> > Regards,
> > Paul Elschot
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org/tech/lucene.asp
>
> Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
> LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_42068_23504604.1179409410340--