mahout-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [CONF] Apache Lucene Mahout > Creating Vectors from Text
Date Sat, 01 Aug 2009 14:40:01 GMT
Space: Apache Lucene Mahout (
Page: Creating Vectors from Text (

Edited by Grant Ingersoll:

h1. Introduction

For clustering documents it is usually necessary to convert the raw text into vectors that
can then be consumed by the clustering [Algorithms].  These approaches are described below.

h1. From Lucene

Mahout has utilities that allow one to easily produce Mahout Vector representations from a
Lucene (and Solr, since they are they same) index.

For this, we assume you know how to build a Lucene/Solr index.  For those who don't, it is
probably easiest to get up and running using [Solr|] as it can
ingest things like PDFs, XML, Office, etc. and create a Lucene index.  For those wanting to
use just Lucene, see the Lucene [website|] or check out _Lucene
In Action_ by Erik Hatcher, Otis Gospodnetic and Mike McCandless.

To get started, make sure you get a fresh copy of Mahout from [SVN|]
and are comfortable building it.  You will also need to [apply the patch|]
on MAHOUT-126.  This patch creates a "utils" module in Mahout at the same level as the Core
that defines utilities for working with Mahout.  In this case, it defines interfaces and implementations
for efficiently iterating over a Data Source (it only supports Lucene currently, but should
be extensible to databases, Solr, etc.) and produces a Mahout Vector file and term dictionary
which can then be used for clustering.   The main code for driving this is the Driver program
located in the org.apache.mahout.utils.vectors package.  The Driver program offers several
input options, which can be displayed by specifying the --help option.  Examples of running
the Driver are included below:

h2. Generating an output file from a Lucene Index

java -cp <CLASSPATH> org.apache.mahout.utils.vectors.lucene.Driver --dir <PATH TO
of vectors to output>> <--norm {INF|integer >= 0}> <--idField <Name of
the idField in the Lucene index>>

h3. Create 50 Vectors from an Index 
org.apache.mahout.utils.vectors.lucene.Driver --dir <PATH>/wikipedia/solr/data/index
--field body --dictOut <PATH>/solr/wikipedia/dict.txt --output <PATH>/solr/wikipedia/out.txt
--max 50
This uses the index specified by --dir and the body field in it and writes out the info to
the output dir and the dictionary to dict.txt.  It only outputs 50 vectors.  If you don't
specify --max, then all the documents in the index are output.

h3. Normalize 50 Vectors from an Index using the [L_2 Norm|]
org.apache.mahout.utils.vectors.lucene.Driver --dir <PATH>/wikipedia/solr/data/index
--field body --dictOut <PATH>/solr/wikipedia/dict.txt --output <PATH>/solr/wikipedia/out.txt
--max 50 --norm 2

h2. Background


h1. From a Database


h1. Other


Change your notification preferences:

View raw message