Return-Path: Delivered-To: apmail-incubator-lucene-net-commits-archive@locus.apache.org Received: (qmail 34627 invoked from network); 8 Jan 2007 04:51:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 8 Jan 2007 04:51:23 -0000 Received: (qmail 54400 invoked by uid 500); 8 Jan 2007 04:51:30 -0000 Delivered-To: apmail-incubator-lucene-net-commits-archive@incubator.apache.org Received: (qmail 54384 invoked by uid 500); 8 Jan 2007 04:51:30 -0000 Mailing-List: contact lucene-net-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: lucene-net-dev@incubator.apache.org Delivered-To: mailing list lucene-net-commits@incubator.apache.org Received: (qmail 54373 invoked by uid 99); 8 Jan 2007 04:51:30 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Jan 2007 20:51:29 -0800 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO eris.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Jan 2007 20:51:18 -0800 Received: by eris.apache.org (Postfix, from userid 65534) id 1FCB11A9829; Sun, 7 Jan 2007 20:49:54 -0800 (PST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r493946 [7/9] - in /incubator/lucene.net/site/docs/2.0: ./ Highlighter.Net/ Highlighter.Net/ndoc_msdn_temp/ Date: Mon, 08 Jan 2007 04:49:48 -0000 To: lucene-net-commits@incubator.apache.org From: aroush@apache.org X-Mailer: svnmailer-1.1.0 Message-Id: <20070108044954.1FCB11A9829@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Added: incubator/lucene.net/site/docs/2.0/Highlighter.Net/Lucene.Net.Highlight.TokenSources.GetAnyTokenStream.html URL: http://svn.apache.org/viewvc/incubator/lucene.net/site/docs/2.0/Highlighter.Net/Lucene.Net.Highlight.TokenSources.GetAnyTokenStream.html?view=auto&rev=493946 ============================================================================== --- incubator/lucene.net/site/docs/2.0/Highlighter.Net/Lucene.Net.Highlight.TokenSources.GetAnyTokenStream.html (added) +++ incubator/lucene.net/site/docs/2.0/Highlighter.Net/Lucene.Net.Highlight.TokenSources.GetAnyTokenStream.html Sun Jan 7 20:49:41 2007 @@ -0,0 +1,66 @@ + + + + + TokenSources.GetAnyTokenStream Method + + + + + +

+ + + + + +

Highlighter.Net 2.0.0

TokenSources.GetAnyTokenStream Method

A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable

public static TokenStream GetAnyTokenStream(
   IndexReader reader,
   int docId,
   string field,
   Analyzer analyzer
);

Parameters

+ reader +: +
+ docId +: +
+ field +: +
+ analyzer +: +

Return Value

null if field not stored correctly

TokenSources.GetTokenStream Method (TermPositionVector)

public static TokenStream GetTokenStream(
TermPositionVector tpv
);

TokenSources.GetTokenStream Method (TermPositionVector, Boolean)

Low level api. Returns a token stream or null if no offset info available in index. This can be used to feed the highlighter with a pre-parsed token stream In my tests the speeds to recreate 1000 token streams using this method are: - with TermVector offset only data stored - 420 milliseconds - with TermVector offset AND position data stored - 271 milliseconds (nb timings for TermVector with position data are based on a tokenizer with contiguous positions - no overlaps or gaps) The cost of not using TermPositionVector to store pre-parsed content and using an analyzer to re-parse the original content: - reanalyzing the original content - 980 milliseconds The re-analyze timings will typically vary depending on - 1) The complexity of the analyzer code (timings above were using a stemmer/lowercaser/stopword combo) 2) The number of other fields (Lucene reads ALL fields off the disk when accessing just one document field - can cost dear!) 3) Use of compression on field s torage - could be faster cos of compression (less disk IO) or slower (more CPU burn) depending on the content.

public static TokenStream GetTokenStream(
TermPositionVector tpv,
bool tokenPositionsGuaranteedContiguous
);

Parameters

+ tpv +: +
+ tokenPositionsGuaranteedContiguous +: true if the token position numbers have no overlaps or gaps. If looking to eek out the last drops of performance, set to true. If in doubt, set to false.

TokenSources.GetTokenStream Method (IndexReader, Int32, String)

public static TokenStream GetTokenStream(
   IndexReader reader,
   int docId,
   string field
);

TokenSources.GetTokenStream Method (IndexReader, Int32, String, Analyzer)

public static TokenStream GetTokenStream(
   IndexReader reader,
   int docId,
   string field,
   Analyzer analyzer
);

TokenSources.GetTokenStream Method

Overload List

+ public static TokenStream GetTokenStream(IndexReader,int,string); +

+ public static TokenStream GetTokenStream(IndexReader,int,string,Analyzer); +

+ public static TokenStream GetTokenStream(TermPositionVector); +

+ public static TokenStream GetTokenStream(TermPositionVector,bool); +

TokenSources.StoredTokenStream.Next Method

public override Token Next();

TokenSources.StoredTokenStream Class

For a list of all members of this type, see TokenSources.StoredTokenStream Members.

+ System.Object +
Lucene.Net.Analysis.TokenStream
Lucene.Net.Highlight.TokenSources.StoredTokenStream

public class TokenSources.StoredTokenStream : TokenStream

Thread Safety

Public static (Shared in Visual Basic) members of this type are + safe for multithreaded operations. Instance members are not guaranteed to be + thread-safe.

Requirements

+ Namespace: + Lucene.Net.Highlight +

+ Assembly: Highlighter.Net (in Highlighter.Net.dll) +

TokenSources.StoredTokenStream Members +

+ TokenSources.StoredTokenStream overview +

Public Instance Methods

+ + + + + + +

Close (inherited from Lucene.Net.Analysis.TokenStream)
Equals (inherited from Object)	+ Determines whether the specified Object is equal to the current Object. +
GetHashCode (inherited from Object)	+ Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. +
GetType (inherited from Object)	+ Gets the Type of the current instance. +
Next
ToString (inherited from Object)	+ Returns a String that represents the current Object. +

Protected Instance Methods

+ + +

Finalize (inherited from Object)	+ Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. +
MemberwiseClone (inherited from Object)	+ Creates a shallow copy of the current Object. +

TokenSources.StoredTokenStream Methods

The methods of the TokenSources.StoredTokenStream class are listed below. For a complete list of TokenSources.StoredTokenStream class members, see the TokenSources.StoredTokenStream Members topic.

Public Instance Methods

+ + + + + + +

Close (inherited from Lucene.Net.Analysis.TokenStream)
Equals (inherited from Object)	+ Determines whether the specified Object is equal to the current Object. +
GetHashCode (inherited from Object)	+ Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. +
GetType (inherited from Object)	+ Gets the Type of the current instance. +
Next
ToString (inherited from Object)	+ Returns a String that represents the current Object. +

Protected Instance Methods

+ + +

Finalize (inherited from Object)	+ Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. +
MemberwiseClone (inherited from Object)	+ Creates a shallow copy of the current Object. +

TokenSources Class

Hides implementation issues associated with obtaining a TokenStream for use with the higlighter - can obtain from TermFreqVectors with offsets and (optionally) positions or from Analyzer class reparsing the stored content.

For a list of all members of this type, see TokenSources Members.

+ System.Object +
Lucene.Net.Highlight.TokenSources

public class TokenSources

Thread Safety

Public static (Shared in Visual Basic) members of this type are + safe for multithreaded operations. Instance members are not guaranteed to be + thread-safe.

Requirements

+ Namespace: + Lucene.Net.Highlight +

+ Assembly: Highlighter.Net (in Highlighter.Net.dll) +

TokenSources Constructor

Initializes a new instance of the TokenSources class.

public TokenSources();

TokenSources Members +

+ TokenSources overview +

Public Static Methods

+ + +

GetAnyTokenStream	A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
GetTokenStream	Overloaded.

Public Instance Constructors

+ + + + + +

+ TokenSources Constructor +

Initializes a new instance of the TokenSources class.

Public Instance Methods

+ + + + +

Equals (inherited from Object)	+ Determines whether the specified Object is equal to the current Object. +
GetHashCode (inherited from Object)	+ Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. +
GetType (inherited from Object)	+ Gets the Type of the current instance. +
ToString (inherited from Object)	+ Returns a String that represents the current Object. +

Protected Instance Methods

+ + +

Finalize (inherited from Object)	+ Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. +
MemberwiseClone (inherited from Object)	+ Creates a shallow copy of the current Object. +

TokenSources Methods

The methods of the TokenSources class are listed below. For a complete list of TokenSources class members, see the TokenSources Members topic.

Public Static Methods

+ + +

GetAnyTokenStream	A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
GetTokenStream	Overloaded.

Public Instance Methods

+ + + + +

Equals (inherited from Object)	+ Determines whether the specified Object is equal to the current Object. +
GetHashCode (inherited from Object)	+ Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. +
GetType (inherited from Object)	+ Gets the Type of the current instance. +
ToString (inherited from Object)	+ Returns a String that represents the current Object. +

Protected Instance Methods

+ + +

Finalize (inherited from Object)	+ Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. +
MemberwiseClone (inherited from Object)	+ Creates a shallow copy of the current Object. +

WeightedTerm.GetTerm Method

public virtual string GetTerm();

Return Value

the term value (stemmed)

WeightedTerm.GetWeight Method

public virtual float GetWeight();

Return Value

the weight associated with this term

WeightedTerm.SetTerm Method

public virtual void SetTerm(
string term
);

Parameters

+ term +: the term value (stemmed)

WeightedTerm.SetWeight Method

public virtual void SetWeight(
float weight
);

Parameters

+ weight +: the weight associated with this term

WeightedTerm Class

Lightweight class to hold term and a weight value used for scoring this term

For a list of all members of this type, see WeightedTerm Members.

+ System.Object +
Lucene.Net.Highlight.WeightedTerm

public class WeightedTerm

Thread Safety

Public static (Shared in Visual Basic) members of this type are + safe for multithreaded operations. Instance members are not guaranteed to be + thread-safe.

Requirements

+ Namespace: + Lucene.Net.Highlight +

+ Assembly: Highlighter.Net (in Highlighter.Net.dll) +

WeightedTerm Constructor

public WeightedTerm(
float weight,
string term
);

WeightedTerm Members +

+ WeightedTerm overview +

Public Instance Constructors

+ + + + + +

+ WeightedTerm Constructor +

Public Instance Methods

+ + + + + + + + +

Equals (inherited from Object)	+ Determines whether the specified Object is equal to the current Object. +
GetHashCode (inherited from Object)	+ Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. +
GetTerm
GetType (inherited from Object)	+ Gets the Type of the current instance. +
GetWeight
SetTerm
SetWeight
ToString (inherited from Object)	+ Returns a String that represents the current Object. +

Protected Instance Methods

+ + +

Finalize (inherited from Object)	+ Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. +
MemberwiseClone (inherited from Object)	+ Creates a shallow copy of the current Object. +

WeightedTerm Methods

The methods of the WeightedTerm class are listed below. For a complete list of WeightedTerm class members, see the WeightedTerm Members topic.

Public Instance Methods

+ + + + + + + + +

Equals (inherited from Object)	+ Determines whether the specified Object is equal to the current Object. +
GetHashCode (inherited from Object)	+ Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. +
GetTerm
GetType (inherited from Object)	+ Gets the Type of the current instance. +
GetWeight
SetTerm
SetWeight
ToString (inherited from Object)	+ Returns a String that represents the current Object. +

Protected Instance Methods

+ + +

Finalize (inherited from Object)	+ Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. +
MemberwiseClone (inherited from Object)	+ Creates a shallow copy of the current Object. +

TokenSources.GetAnyTokenStream Method

Parameters

Return Value

See Also

TokenSources.GetTokenStream Method (TermPositionVector)

See Also

TokenSources.GetTokenStream Method (TermPositionVector, Boolean)

Parameters

See Also

TokenSources.GetTokenStream Method (IndexReader, Int32, String)

See Also

TokenSources.GetTokenStream Method (IndexReader, Int32, String, Analyzer)

See Also

TokenSources.GetTokenStream Method

Overload List

See Also

TokenSources.StoredTokenStream.Next Method

See Also

TokenSources.StoredTokenStream Class

Thread Safety

Requirements

See Also

TokenSources.StoredTokenStream Members +

Public Instance Methods

Protected Instance Methods

See Also

TokenSources.StoredTokenStream Methods

Public Instance Methods

Protected Instance Methods

See Also

TokenSources Class

Thread Safety

Requirements

See Also

TokenSources Constructor

See Also

TokenSources Members +

Public Static Methods

Public Instance Constructors

Public Instance Methods

Protected Instance Methods

See Also

TokenSources Methods

Public Static Methods

Public Instance Methods

Protected Instance Methods

See Also

WeightedTerm.GetTerm Method

Return Value

See Also

WeightedTerm.GetWeight Method

Return Value

See Also

WeightedTerm.SetTerm Method

Parameters

See Also

WeightedTerm.SetWeight Method

Parameters

See Also

WeightedTerm Class

Thread Safety

Requirements

See Also

WeightedTerm Constructor

See Also

WeightedTerm Members +

Public Instance Constructors

Public Instance Methods

Protected Instance Methods

See Also

WeightedTerm Methods

Public Instance Methods

Protected Instance Methods

See Also