lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "AJ Weber" <awe...@comcast.net>
Subject Re: replace field in doc?
Date Tue, 15 Apr 2008 19:04:38 GMT
Characters or "terms"?  (And btw: what's the difference?)  The javadoc says 10,000 "terms",
which I assume generally equates to "words" (and given that the analyzer might use stemming,
stop words, etc.).

Great info.  Thanks again!

-AJ
  ----- Original Message ----- 
  From: Erick Erickson 
  To: java-user@lucene.apache.org ; AJ Weber 
  Sent: Tuesday, April 15, 2008 2:50 PM
  Subject: Re: replace field in doc?


  Well, "my way" would certainly be simpler to read six months from
  now when you look at this code again <G>........

  And I'm quite sure you can add the same field multiple times, so 
  whatever you want.

  Do note, though, that Lucene defaults to 10,000 characters in any
  single field no matter which way you build it up, so you might need to
  be aware of IndexWriter.setMaxFieldLength().

  Good Luck!
  Erick



  On Tue, Apr 15, 2008 at 1:39 PM, AJ Weber <aweber@comcast.net> wrote:

    I ended up doing this:

                               String docText = doc.get("body");
                               Field fCurAll = doc.getField("all");
                               if ((fCurAll != null) && (docText != null)) {
                                   String newAll = fCurAll.stringValue() + " " + docText;
                                   logCat.debug("Updating 'all' field with props + bodyText");
                                   fCurAll.setValue(newAll);
                               }
                               else if ((fCurAll == null) && (docText != null))
                                   doc.add(new Field("all", docText, Field.Store.NO, Field.Index.TOKENIZED));

    But I suppose if what you say is true, it would be more efficient to do it "your way",
because I don't need to chew-up memory and CPU by extracting the existing info, concat it,
then put it back, right?

    (Since this is the "all" field, for 'global searching', and I'm not storing it anyway,
I'm not concerned about the positions too much.)

    Thanks a bunch for the quick reply!
    -AJ

     ----- Original Message -----
     From: Erick Erickson
     To: java-user@lucene.apache.org ; AJ Weber
     Sent: Tuesday, April 15, 2008 1:12 PM
     Subject: Re: replace field in doc?


     You can freely add the same field (with different text) to a doc. For
     instance
     Document doc = new Document();
     doc.add("field", "this is the first");
     doc.add("field", "starting the second ");
     IndexWriter.addDocument(doc)

     is functionally the same as


     Document doc = new Document();
     doc.add("field", "this is the first starting the second");
     IndexWriter.addDocument(doc)


     with one subtle difference. The first version will add
     whatever is returned from Analyzer.getPositionIncrementGap
     to the term position of the first token in adds 2-n. That is,
     say getPositionIncrementGap returned 100. Then
     the token "first" would have position 3 and "starting"
     would have position 103 (I may be off by one on both these,
     but you get the idea).

     The default return is 1, so if you do nothing special, the two
     calls will be identical.

     Why would you return something other than 1 from
     getPositionIncrementGap? Well, if you want to dance
     fancy and, say, NOT span across the two lines above
     for, say, SpanNear clauses, you could return a large
     number that was bigger than the max allowable span.

     Best
     Erick

     On Tue, Apr 15, 2008 at 10:26 AM, AJ Weber <aweber@comcast.net> wrote:

     > I'm curious how people are building the "all" Field (for searching "all of
     > the terms at once").
     >
     > I understand using store=NO, Index=Tokenized is generally the way to add
     > the field, but what if I need to basically use multiple classes to build my
     > Document before adding it to the index (keeping things modular and
     > reusable)?
     >
     > If one class adds the Field (i.e. doc.add("all", sCurrentString,
     > Field.Store.NO, Field.Index.TOKENIZED) ), then I pass the document ("doc"
     > in this example) to another class, how do I basically concat any additional
     > field-values/strings to the existing text-Field?  This is all before the
     > Document is added to the index in the first-place.
     >
     > I'm concerned that doing a doc.get("all") and then adding it back to the
     > doc with the additional string will result in two fields of the name "all"
     > (I seem to have read this is the functionality somewhere)???
     >
     > Thanks in advance!
     > -AJ



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message