From java-dev-return-9639-apmail-lucene-java-dev-archive=lucene.apache.org@lucene.apache.org Wed Mar 23 01:45:57 2005 Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 79771 invoked from network); 23 Mar 2005 01:45:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 23 Mar 2005 01:45:57 -0000 Received: (qmail 81655 invoked by uid 500); 23 Mar 2005 01:45:56 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 81212 invoked by uid 500); 23 Mar 2005 01:45:55 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 81198 invoked by uid 500); 23 Mar 2005 01:45:55 -0000 Delivered-To: apmail-jakarta-lucene-dev@jakarta.apache.org Received: (qmail 81195 invoked by uid 99); 23 Mar 2005 01:45:55 -0000 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=NO_REAL_NAME X-Spam-Check-By: apache.org Received: from ajax-1.apache.org (HELO ajax.apache.org) (192.87.106.226) by apache.org (qpsmtpd/0.28) with ESMTP; Tue, 22 Mar 2005 17:45:54 -0800 Received: from ajax.apache.org (ajax.apache.org [127.0.0.1]) by ajax.apache.org (8.12.11/8.12.11) with ESMTP id j2N1jp1O030611 for ; Wed, 23 Mar 2005 02:45:51 +0100 Received: (from nobody@localhost) by ajax.apache.org (8.12.11/8.12.11/Submit) id j2N1jonL030607; Wed, 23 Mar 2005 02:45:50 +0100 Date: Wed, 23 Mar 2005 02:45:50 +0100 Message-Id: <200503230145.j2N1jonL030607@ajax.apache.org> From: bugzilla@apache.org To: lucene-dev@jakarta.apache.org Subject: DO NOT REPLY [Bug 34066] - [PATCH] Extension to binary Fields that allows fixed byte buffer X-Bugzilla-Reason: AssignedTo X-Virus-Checked: Checked X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG· RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT . ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND· INSERTED IN THE BUG DATABASE. http://issues.apache.org/bugzilla/show_bug.cgi?id=34066 ------- Additional Comments From chuck@manawiz.com 2005-03-23 02:45 ------- I don't see how the first change (initializing fieldsDataLength to -1 and testing it in binaryLength()) provides any additional safety. The patch always initializes fieldsDataLength in Field() for any binary value, so unless that was changed the value would never be -1 in binaryLength(). The main risk in this area of the code is that somebody calls binaryValue() without calling binaryLength() to get the length, which seems impossible to address. Another risk is that somebody calls binaryLength() on a non-binary field -- the current patch returns null in that case, consistent with binaryValue(), while the changed version would get an NPE. I'm probably missing some other case that you see. The second change (validating the length passed to Field()) seems an improvement. But the biggest risk with this patch is the one I outlined in the Javadoc, i.e. that somebody passes the same byte array in two different calls to Field before the use in the first call is consumed (e.g., 2 fields in the same Document, or the same field in 2 Documents before either is indexed). I don't see a good way to protect against that (without a performance hit). The patch is a bit risky due to this last consideration, but the performance gain is substantial in the particular case where it is necessary to store large document bodies in Lucene. I'm indexing with this now and doing zero allocations by using nio ByteBuffer and CharBuffer views on a fixed byte array that holds the successive document bodies. It rips. I'm using compression outside of Lucene (using the same zlib). The combination of outside-compression and the patch reduces 5 allocations of each document body to 0, when compared to passing the bodies as text and letting Lucene do the compression (the biggest part of this is the outside-compression). I don't object to either change and would be happy to see them if it means this gets committed so I can eliminate my local patch! Chuck -- Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org