lucene-solr-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Solr Wiki] Trivial Update of "OpenNLP" by LanceXNorskog
Date Thu, 05 Jul 2012 00:55:15 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.

The "OpenNLP" page has been changed by LanceXNorskog:
http://wiki.apache.org/solr/OpenNLP?action=diff&rev1=2&rev2=3

+ <!> [[Solr4.0]] <<TableOfContents(3)>>
- <!> [[Solr4.0]]
- <<TableOfContents(3)>>
  
- <!> This page discusses uncommitted code and design.  See [[https://issues.apache.org/jira/browse/LUCENE-2899|LUCENE-2899]]
for the main JIRA issue tracking this development. The issue is packaged as a Solr contrib,
but will be split between Lucene and Solr. There is some design work needed before this can
be committed.
+ <!> This page discusses uncommitted code and design.  See [[https://issues.apache.org/jira/browse/LUCENE-2899|LUCENE-2899]]
for the main JIRA issue tracking this development. The issue is packaged as a Solr contrib,
but is split between Lucene and Solr.
  
- NLP is a large field of inquiry. Unless you are familiar with it you may find this patch
confusing.
+ NLP is a large field of inquiry. Unless you are familiar with it you may find this patch
confusing. The [[http://opennlp.apache.org/|Apache OpenNLP project]] is the best place to
learn what this package can do.
  
  == Introduction ==
- 
  OpenNLP is a toolkit for Natural Language Processing (NLP). It is an Apache top-level project
located [[here|http://opennlp.apache.org/]]. It includes implementations of many popular NLP
algorithms. This project integrates some of its features into Lucene and Solr. This first
effort incorporates Analyzer chain tools for sentence detection, tokenization, Parts-of-Speech
tagging (nouns, verbs, ejaculations, etc.), Chunking (noun phrases, verb phrases) and Named
Entity Recognition.  See the OpenNLP project page for information on the implementations.
 Here are some use cases:
  
  === Indexing interesting words ===
@@ -18, +16 @@

  Chunking lets you create N-Grams only within noun and verb phrases.
  
  === Named Entity Recognition ===
- Named Entity Recognition identifies names, dates, places, currency and other types of data
within free text. This is profoundly useful in searching. Or, you can create autosuggest entries
with icons for 'Name', 'Place', etc.
+ Named Entity Recognition identifies names, dates, places, currency and other types of data
within free text. This is profoundly useful in searching. Or, you can create facets or autosuggest
entries with icons for 'Name', 'Place', etc.
  
  == Analyzer tools ==
- 
  The OpenNLP Tokenizer behavior is similar to the WhiteSpaceTokenizer but is smart about
inter-word punctuation. The term stream looks very much like the way you parse words and punctuation
while reading. The OpenNLP taggers assign payloads to terms. There are tools to filter the
term stream according to the payload values, and to remove the payloads.
  
  === solr.OpenNLPTokenizerFactory ===
- 
  Tokenizes text into sentences or words.
  
  This Tokenizer uses the OpenNLP Sentence Detector and/or Tokenizer classes. When used together,
the Tokenizer receives sentences and can do a better job. The arguments give the file names
of the statistical models:
@@ -40, +36 @@

        </analyzer>
      </fieldType>
  }}}
- 
  === solr.OpenNLPFilterFactory ===
- 
- Tags words using one or more technologies: Parts-of-Speech, Chunking, and Named Entity Recognition.

+ Tags words using one or more technologies: Parts-of-Speech, Chunking, and Named Entity Recognition.
  
  {{{
      <fieldType name="text_opennlp_pos" class="solr.TextField" positionIncrementGap="100">
@@ -51, +45 @@

          <tokenizer class="solr.OpenNLPTokenizerFactory"
            tokenizerModel="opennlp/en-token.bin"
          />
-         <filter class="solr.OpenNLPFilterFactory" 
+         <filter class="solr.OpenNLPFilterFactory"
            posTaggerModel="opennlp/en-pos-maxent.bin"
-         />       
+         />
        </analyzer>
      </fieldType>
  }}}
- 
  This example assigns parts of speech tags based on a model derived with the [[http://opennlp.apache.org/documentation/1.5.2-incubating/apidocs/opennlp-maxent/index.html|OpenNLP
Maximum Entropy]] implementation. See [[http://opennlp.apache.org/documentation/1.5.2-incubating/manual/opennlp.html#tools.postagger.tagging|OpenNLP
Tagging]] for more information. The tags are from the [[http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html|Penn
Treebank]] tagset
  
  === solr.FilterPayloadsFilterFactory ===
- 
  Filter terms for certain payload values. In this example, retain only terms which have been
marked 'nouns' and 'verbs' with the [[http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html|Penn
Treebank]] tagset.
  
  {{{
          <filter class="solr.FilterPayloadsFilterFactory" keepPayloads="true"
            payloadList="NN,NNS,NNP,NNPS,VB,VBD,VBG,VBN,VBP,VBZ,FW"/>
  }}}
- 
  === solr.StripPayloadsFilterFactory ===
- 
  Remove payloads from terms.
  
  {{{
          <filter class="solr.StripPayloadsFilterFactory"/>
  }}}
+ == Full Example ==
+ This "Noun-Verb Filter" field type assigns parts of speech, retains only nouns and verbs,
and removes the payloads. Free-text search sites (for example, newspaper and magazine articles)
may benefit from this.
  
- == Full Example ==
- 
- This "Noun-Verb Filter" field type assigns parts of speech, retains only nouns and verbs,
and removes the payloads. Free-text search sites (for example, newspaper and magazine articles)
may benefit from this.
  {{{
      <fieldType name="text_opennlp_nvf" class="solr.TextField" positionIncrementGap="100">
        <analyzer>
@@ -94, +83 @@

        </analyzer>
      </fieldType>
  }}}
- 
- This example should work well with most English-language free text. 
+ This example should work well with most English-language free text.
  
  == Installation ==
- 
  See the patch for more information. The short story is you have to download statistical
models from sourceforge to make OpenNLP work- the models do not have an Apache-compatible
license.
  

Mime
View raw message