mahout-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r909887 - in /websites/staging/mahout/trunk/content: ./ users/basics/creating-vectors-from-text.html
Date Sun, 25 May 2014 11:40:52 GMT
Author: buildbot
Date: Sun May 25 11:40:52 2014
New Revision: 909887

Log:
Staging update by buildbot for mahout

Modified:
    websites/staging/mahout/trunk/content/   (props changed)
    websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html

Propchange: websites/staging/mahout/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sun May 25 11:40:52 2014
@@ -1 +1 @@
-1596770
+1597416

Modified: websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html
==============================================================================
--- websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html (original)
+++ websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html Sun
May 25 11:40:52 2014
@@ -241,8 +241,7 @@
 <p><a name="CreatingVectorsfromText-Introduction"></a></p>
 <h1 id="introduction">Introduction</h1>
 <p>For clustering and classifying documents it is usually necessary to convert the
raw text
-into vectors that can then be consumed by the clustering <a href="algorithms.html">Algorithms</a>
-.  These approaches are described below.</p>
+into vectors that can then be consumed by the clustering <a href="algorithms.html">Algorithms</a>.
 These approaches are described below.</p>
 <p><a name="CreatingVectorsfromText-FromLucene"></a></p>
 <h1 id="from-lucene">From Lucene</h1>
 <p><em>NOTE: Your Lucene index must be created with the same version of Lucene
@@ -260,13 +259,13 @@ index.  For those wanting to use just Lu
 McCandless.</p>
 <p>To get started, make sure you get a fresh copy of Mahout from <a href="http://mahout.apache.org/developers/buildingmahout.html">SVN</a>
  and are comfortable building it. It defines interfaces and implementations
-for efficiently iterating over a Data Source (it only supports Lucene
+for efficiently iterating over a data source (it only supports Lucene
 currently, but should be extensible to databases, Solr, etc.) and produces
 a Mahout Vector file and term dictionary which can then be used for
-clustering.   The main code for driving this is the Driver program located
-in the org.apache.mahout.utils.vectors package.  The Driver program offers
+clustering.   The main code for driving this is the driver program located
+in the org.apache.mahout.utils.vectors package.  The driver program offers
 several input options, which can be displayed by specifying the --help
-option.  Examples of running the Driver are included below:</p>
+option.  Examples of running the driver are included below:</p>
 <p><a name="CreatingVectorsfromText-GeneratinganoutputfilefromaLuceneIndex"></a></p>
 <h4 id="generating-an-output-file-from-a-lucene-index">Generating an output file from
a Lucene Index</h4>
 <div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span
class="o">/</span><span class="n">bin</span><span class="o">/</span><span
class="n">mahout</span> <span class="n">lucene</span><span class="p">.</span><span
class="n">vector</span> 
@@ -324,12 +323,12 @@ option.  Examples of running the Driver 
 </pre></div>
 
 
-<h4 id="create-50-vectors-from-an-index">Create 50 Vectors from an Index</h4>
+<h4 id="example-create-50-vectors-from-an-index">Example: Create 50 Vectors from an
Index</h4>
 <div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span
class="o">/</span><span class="n">bin</span><span class="o">/</span><span
class="n">mahout</span> <span class="n">lucene</span><span class="p">.</span><span
class="n">vector</span>
-    <span class="o">--</span><span class="n">dir</span> <span
class="o">&lt;</span><span class="n">PATH</span><span class="o">&gt;/</span><span
class="n">wikipedia</span><span class="o">/</span><span class="n">solr</span><span
class="o">/</span><span class="n">data</span><span class="o">/</span><span
class="n">index</span> 
+    <span class="o">--</span><span class="n">dir</span> $<span
class="n">WORK_DIR</span><span class="o">/</span><span class="n">wikipedia</span><span
class="o">/</span><span class="n">solr</span><span class="o">/</span><span
class="n">data</span><span class="o">/</span><span class="n">index</span>

     <span class="o">--</span><span class="n">field</span> <span
class="n">body</span> 
-    <span class="o">--</span><span class="n">dictOut</span> <span
class="o">&lt;</span><span class="n">PATH</span><span class="o">&gt;/</span><span
class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span
class="o">/</span><span class="n">dict</span><span class="p">.</span><span
class="n">txt</span>
-    <span class="o">--</span><span class="n">output</span> <span
class="o">&lt;</span><span class="n">PATH</span><span class="o">&gt;/</span><span
class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span
class="o">/</span><span class="n">out</span><span class="p">.</span><span
class="n">txt</span> 
+    <span class="o">--</span><span class="n">dictOut</span> $<span
class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span
class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span
class="n">dict</span><span class="p">.</span><span class="n">txt</span>
+    <span class="o">--</span><span class="n">output</span> $<span
class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span
class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span
class="n">out</span><span class="p">.</span><span class="n">txt</span>

     <span class="o">--</span><span class="n">max</span> 50
 </pre></div>
 
@@ -339,12 +338,12 @@ out the info to the output dir and the d
 outputs 50 vectors.  If you don't specify --max, then all the documents in
 the index are output.</p>
 <p><a name="CreatingVectorsfromText-50VectorsFromLuceneL2Norm"></a></p>
-<h4 id="creating-50-normalized-vectors-from-a-lucene-index-using-the-l_2-norm">Creating
50 Normalized Vectors from a Lucene Index using the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2
Norm</a></h4>
+<h4 id="example-creating-50-normalized-vectors-from-a-lucene-index-using-the-l_2-norm">Example:
Creating 50 Normalized Vectors from a Lucene Index using the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2
Norm</a></h4>
 <div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span
class="o">/</span><span class="n">bin</span><span class="o">/</span><span
class="n">mahout</span> <span class="n">lucene</span><span class="p">.</span><span
class="n">vector</span> 
-    <span class="o">--</span><span class="n">dir</span> <span
class="o">&lt;</span><span class="n">PATH</span><span class="o">&gt;/</span><span
class="n">wikipedia</span><span class="o">/</span><span class="n">solr</span><span
class="o">/</span><span class="n">data</span><span class="o">/</span><span
class="n">index</span> 
+    <span class="o">--</span><span class="n">dir</span> $<span
class="n">WORK_DIR</span><span class="o">/</span><span class="n">wikipedia</span><span
class="o">/</span><span class="n">solr</span><span class="o">/</span><span
class="n">data</span><span class="o">/</span><span class="n">index</span>

     <span class="o">--</span><span class="n">field</span> <span
class="n">body</span> 
-    <span class="o">--</span><span class="n">dictOut</span> <span
class="o">&lt;</span><span class="n">PATH</span><span class="o">&gt;/</span><span
class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span
class="o">/</span><span class="n">dict</span><span class="p">.</span><span
class="n">txt</span>
-    <span class="o">--</span><span class="n">output</span> <span
class="o">&lt;</span><span class="n">PATH</span><span class="o">&gt;/</span><span
class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span
class="o">/</span><span class="n">out</span><span class="p">.</span><span
class="n">txt</span> 
+    <span class="o">--</span><span class="n">dictOut</span> $<span
class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span
class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span
class="n">dict</span><span class="p">.</span><span class="n">txt</span>
+    <span class="o">--</span><span class="n">output</span> $<span
class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span
class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span
class="n">out</span><span class="p">.</span><span class="n">txt</span>

     <span class="o">--</span><span class="n">max</span> 50 
     <span class="o">--</span><span class="n">norm</span> 2
 </pre></div>
@@ -355,7 +354,7 @@ the index are output.</p>
 <p>Mahout has utilities to generate Vectors from a directory of text
 documents. Before creating the vectors, you need to convert the documents
 to SequenceFile format. SequenceFile is a hadoop class which allows us to
-write arbitary key,value pairs into it. The DocumentVectorizer requires the
+write arbitary (key, value) pairs into it. The DocumentVectorizer requires the
 key to be a Text with a unique document id, and value to be the Text
 content in UTF-8 format.</p>
 <p>You may find <a href="http://tika.apache.org/">Tika</a> helpful in converting
@@ -363,9 +362,7 @@ binary documents to text.</p>
 <p><a name="CreatingVectorsfromText-ConvertingdirectoryofdocumentstoSequenceFileformat"></a></p>
 <h4 id="converting-directory-of-documents-to-sequencefile-format">Converting directory
of documents to SequenceFile format</h4>
 <p>Mahout has a nifty utility which reads a directory path including its
-sub-directories and creates the SequenceFile in a chunked manner for us.
-the document id generated is <PREFIX><RELATIVE PATH FROM
-PARENT>/document.txt</p>
+sub-directories and creates the SequenceFile in a chunked manner for us.</p>
 <div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span
class="o">/</span><span class="n">bin</span><span class="o">/</span><span
class="n">mahout</span> <span class="n">seqdirectory</span> 
     <span class="o">--</span><span class="n">input</span> <span
class="p">(</span><span class="o">-</span><span class="nb">i</span><span
class="p">)</span> <span class="n">input</span>                     
 <span class="n">Path</span> <span class="n">to</span> <span class="n">job</span>
<span class="n">input</span> <span class="n">directory</span><span
class="p">.</span>   
     <span class="o">--</span><span class="n">output</span> <span
class="p">(</span><span class="o">-</span><span class="n">o</span><span
class="p">)</span> <span class="n">output</span>                    
<span class="n">The</span> <span class="n">directory</span> <span
class="n">pathname</span> <span class="k">for</span>     
@@ -395,10 +392,11 @@ PARENT>/document.txt</p>
     <span class="o">--</span><span class="n">help</span> <span
class="p">(</span><span class="o">-</span><span class="n">h</span><span
class="p">)</span>                              <span class="n">Print</span>
<span class="n">out</span> <span class="n">help</span>           
     
     <span class="o">--</span><span class="n">tempDir</span> <span
class="n">tempDir</span>                        <span class="n">Intermediate</span>
<span class="n">output</span> <span class="n">directory</span>  
     <span class="o">--</span><span class="n">startPhase</span> <span
class="n">startPhase</span>                  <span class="n">First</span>
<span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
            
-    <span class="o">--</span><span class="n">endPhase</span> <span
class="n">endPhase</span>                      <span class="n">Last</span>
<span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
 <span class="o">&gt;</span>
+    <span class="o">--</span><span class="n">endPhase</span> <span
class="n">endPhase</span>                      <span class="n">Last</span>
<span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
 </pre></div>
 
 
+<p>The output of seqDirectory will be a Sequence file &lt; Text, Text &gt;
of all documents (/sub-directory-path/documentFileName, documentText).</p>
 <p><a name="CreatingVectorsfromText-CreatingVectorsfromSequenceFile"></a></p>
 <h4 id="creating-vectors-from-sequencefile">Creating Vectors from SequenceFile</h4>
 <p>From the sequence file generated from the above step run the following to
@@ -455,8 +453,35 @@ generate vectors. </p>
 </pre></div>
 
 
-<p>--minSupport is the min frequency for the word to be considered as a feature. --minDF
is the min number of documents the word needs to be in --maxDFPercent is the max value of
the expression (document frequency of a word/total number of document) to be considered as
good feature to be in the document. These options are helpful in removing high frequency features
like stop words.
-<a name="CreatingVectorsfromText-Background"></a></p>
+<p>This will create SequenceFiles of tokenized documents &lt; Text, StringTuple
&gt;  (docID, tokenizedDoc) and vectorized documents &lt; Text, VectorWritable &gt;
(docID, TF-IDF Vector).  </p>
+<p>As well, seq2sparse will create SequenceFiles for: a dictionary (wordIndex, word),
a word frequency count (wordIndex, count) and a document frequency count (wordIndex, DFCount)
in the output directory. </p>
+<p>The --minSupport option is the min frequency for the word to be considered as a
feature; --minDF is the min number of documents the word needs to be in; --maxDFPercent is
the max value of the expression (document frequency of a word/total number of document) to
be considered as good feature to be in the document. These options are helpful in removing
high frequency features like stop words.</p>
+<p>The vectorized documents can then be used as input to many of Mahout's classification
and clustering algorithms.</p>
+<h4 id="example-creating-normalized-tf-idf-vectors-from-a-directory-of-text-documents-using-trigrams-and-the-l_2-norm">Example:
Creating Normalized <a href="http://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>
Vectors from a directory of text documents using <a href="http://en.wikipedia.org/wiki/N-gram">trigrams</a>
and the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2 Norm</a></h4>
+<p>Create sequence files from the directory of text documents:</p>
+<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span
class="o">/</span><span class="n">bin</span><span class="o">/</span><span
class="n">mahout</span> <span class="n">seqdirectory</span> 
+    <span class="o">-</span><span class="nb">i</span> $<span class="n">WORK_DIR</span><span
class="o">/</span><span class="n">reuters</span> 
+    <span class="o">-</span><span class="n">o</span> $<span class="n">WORK_DIR</span><span
class="o">/</span><span class="n">reuters</span><span class="o">-</span><span
class="n">seqdir</span> 
+    <span class="o">-</span><span class="n">c</span> <span class="n">UTF</span><span
class="o">-</span>8
+    <span class="o">-</span><span class="n">chunk</span> 64
+    <span class="o">-</span><span class="n">xm</span> <span class="n">sequential</span>
+</pre></div>
+
+
+<p>Vectorize the documents using trigrams, L_2 length normalization and a maximum document
frequency cutoff of 85%.</p>
+<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span
class="o">/</span><span class="n">bin</span><span class="o">/</span><span
class="n">mahout</span> <span class="n">seq2sparse</span> 
+    <span class="o">-</span><span class="nb">i</span> $<span class="n">WORK_DIR</span><span
class="o">/</span><span class="n">reuters</span><span class="o">-</span><span
class="n">out</span><span class="o">-</span><span class="n">seqdir</span><span
class="o">/</span> 
+    <span class="o">-</span><span class="n">o</span> $<span class="n">WORK_DIR</span><span
class="o">/</span><span class="n">reuters</span><span class="o">-</span><span
class="n">out</span><span class="o">-</span><span class="n">seqdir</span><span
class="o">-</span><span class="n">sparse</span><span class="o">-</span><span
class="n">kmeans</span> 
+    <span class="o">--</span><span class="n">namedVec</span>
+    <span class="o">-</span><span class="n">wt</span> <span class="n">tfidf</span>
+    <span class="o">-</span><span class="n">ng</span> 3
+    <span class="o">-</span><span class="n">n</span> 2
+    <span class="o">--</span><span class="n">maxDFPercent</span>
85
+</pre></div>
+
+
+<p>The sequence file in the $WORK_DIR/reuters-out-seqdir-sparse-kmeans/tfidf-vectors
directory can now be used as input to the Mahout <a href="http://mahout.apache.org/users/clustering/k-means-clustering.html">k-Means</a>
clustering algorithm.</p>
+<p><a name="CreatingVectorsfromText-Background"></a></p>
 <h2 id="background">Background</h2>
 <ul>
 <li><a href="http://markmail.org/thread/l5zi3yk446goll3o">Discussion on centroid
calculations with sparse vectors</a></li>
@@ -469,7 +494,11 @@ question arises of how to convert the ve
 format. Probably the easiest way to go would be to implement your own
 Iterable<Vector> (called VectorIterable in the example below) and then
 reuse the existing VectorWriter classes:</p>
-<div class="codehilite"><pre><span class="n">VectorWriter</span>
<span class="n">vectorWriter</span> <span class="p">=</span> <span
class="n">SequenceFile</span><span class="p">.</span><span class="n">createWriter</span><span
class="p">(</span><span class="n">filesystem</span><span class="p">,</span>
<span class="n">configuration</span><span class="p">,</span> <span
class="n">outfile</span><span class="p">,</span> <span class="n">LongWritable</span><span
class="p">.</span><span class="n">class</span><span class="p">,</span>
<span class="n">SparseVector</span><span class="p">.</span><span
class="n">class</span><span class="p">);</span>
+<div class="codehilite"><pre><span class="n">VectorWriter</span>
<span class="n">vectorWriter</span> <span class="p">=</span> <span
class="n">SequenceFile</span><span class="p">.</span><span class="n">createWriter</span><span
class="p">(</span><span class="n">filesystem</span><span class="p">,</span>
+                                                      <span class="n">configuration</span><span
class="p">,</span>
+                                                      <span class="n">outfile</span><span
class="p">,</span>
+                                                      <span class="n">LongWritable</span><span
class="p">.</span><span class="n">class</span><span class="p">,</span>
+                                                      <span class="n">SparseVector</span><span
class="p">.</span><span class="n">class</span><span class="p">);</span>
 
 <span class="n">long</span> <span class="n">numDocs</span> <span
class="p">=</span> <span class="n">vectorWriter</span><span class="p">.</span><span
class="n">write</span><span class="p">(</span><span class="n">new</span>
<span class="n">VectorIterable</span><span class="p">(),</span> <span
class="n">Long</span><span class="p">.</span><span class="n">MAX_VALUE</span><span
class="p">);</span>
 </pre></div>



Mime
View raw message