Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <5974576.1153191074759.JavaMail.jira@brutus>
Date: Mon, 17 Jul 2006 19:51:14 -0700 (PDT)
From: "Yonik Seeley (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-629) Performance improvement for merging
 stored, compressed fields
In-Reply-To: <25555059.1153172954315.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/LUCENE-629?page=comments#action_12421770 ] 
            
Yonik Seeley commented on LUCENE-629:
-------------------------------------

Thanks Michael, very impressive results!
It might take a little while to review the patch, but the goals and general approach certainly seem correct.

What is the impact (if any) for non-compressed fields?  

> Performance improvement for merging stored, compressed fields
> -------------------------------------------------------------
>
>                 Key: LUCENE-629
>                 URL: http://issues.apache.org/jira/browse/LUCENE-629
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Priority: Minor
>         Attachments: optimized_merge_compressed_fields.patch
>
>
> Hello everyone,
> currently the merging of stored, compressed fields is not optimal for the following reason: every time a stored, compressed field is being merged, the FieldsReader uncompresses the data, hence the FieldsWriter has to compress it again when it writes the merged fields data (.fdt) file. The uncompress/compress step is unneccessary and slows down the merge performance significantly.
> This patch improves the merge performance by avoiding the uncompress/compress step. In the following I give an overview of the changes I made:
>    * Added a new FieldSelectorResult constant named "LOAD_FOR_MERGE" to org.apache.lucene.document.FieldSelectorResult
>    * SegmentMerger now uses an FieldSelector to get stored fields from the FieldsReader. This FieldSelector's accept() method returns the FieldSelectorResult "LOAD_FOR_MERGE" for every field.
>    * Added a new inner class to FieldsReader named "FieldForMerge", which extends  org.apache.lucene.document.AbstractField. This class holds the field properties and its data. If a field has the FieldSelectorResult "LOAD_FOR_MERGE", then the FieldsReader creates an instance of "FieldForMerge" and does not uncompress the field's data.
>    * FieldsWriter checks if the field it is about to write is an instanceof FieldsReader.FieldForMerge. If true, then it does not compress the field data.
> To test the performance I index about 350,000 text files and store the raw text in a stored, compressed field in the lucene index. I use a merge factor of 10. The final index has a size of 366MB. After building the index, I optimize it to measure the pure merge performance.
> Here are the performance results:
> old version:
>    * Time for Indexing:  36.7 minutes
>    * Time for Optimizing: 4.6 minutes
> patched version:
>    * Time for Indexing:  20.8 minutes
>    * Time for Optimizing: 0.5 minutes
> The results show that the index build time improved by about 43%, and the optimizing step is more than 8x faster. 
> A diff of the final indexes (old and patched version) shows, that they are identical. Furthermore, all junit testcases succeeded with the patched version. 
> Regards,
>   Michael Busch

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org