Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: vector generation
From: Grant Ingersoll <gsingers@apache.org>
In-Reply-To: 
 <B648F67990B15146B8646584D20B8DD10B8C0466@TVACOCXVS2.main.tva.gov>
Date: Tue, 24 Nov 2009 11:32:11 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <AFB4A63C-A80B-4534-85AA-395AAFB68449@apache.org>
References: <B648F67990B15146B8646584D20B8DD10B8C0466@TVACOCXVS2.main.tva.gov>
To: mahout-user@lucene.apache.org


On Nov 24, 2009, at 10:32 AM, Patterson, Josh wrote:

> While reading through the wiki and article material on mahout, I =
noticed
> that there was a pre-generation step where vectors were being =
generated
> from either text with Lucene or ARFF with
> org.apache.mahout.utils.vectorsarff.driver.java; Looking at the =
k-means
> driver and mapper (KMeansMapper.java) I noticed that the mapper is
> taking a key and then a Vector (point) as input.
>=20
>=20
>=20
> Would it be smart or practical to make a special record reader for =
your
> file format that read your data in as vectors directly and emitted
> vectors to the mapper in order to skip the pre-generation step? Just
> curious about that, maybe I'm missing something there, or =
vectorization
> would be cumbersome in that position, etc.

Probably would be useful.  No one has taken the steps yet.=20

>=20
>=20
>=20
> Also, in Grant's article on Mahout he includes the vectorized 2.5 GB
> file from Wikipedia that is in the correct format via Lucene to work
> with a Mahout clustering algorithm; Is there a smaller (sub 100 meg)
> version of this that I could play around with? I'm working with basic
> building blocks right now and figuring out the facets of vectorization
> with respect to Mahout so we can learn the base case  (lucene vectors)
> and then move on to our specific case (sensor time series data).

Here's what I did:
Using Solr, create an index, make sure you turn on term vectors for the =
appropriate fields.
Point the Lucene Driver at the index and create the vectors. =20

You could do this even using the Solr tutorial (solr/example) which =
would give you an index of about 20 docs.


Here's the schema.xml I used (or, at least the relevant field =
definitions):
<field name=3D"docid" type=3D"string" indexed=3D"true" stored=3D"true" =
required=3D"true"/>
        <field name=3D"file" type=3D"string" indexed=3D"true" =
stored=3D"true" />

        <field name=3D"doctitle" type=3D"text" indexed=3D"true" =
stored=3D"true" multiValued=3D"true" termVectors=3D"true"/>
        <field name=3D"body" type=3D"text" indexed=3D"true" =
stored=3D"true" multiValued=3D"true" termVectors=3D"true"/>
        <field name=3D"docdate" type=3D"date" indexed=3D"true" =
stored=3D"true" multiValued=3D"false"/>


        <field name=3D"titleBody" type=3D"text" indexed=3D"true" =
stored=3D"false" multiValued=3D"true" termVectors=3D"true"/>

        <field name=3D"spell" type=3D"text_spell" indexed=3D"true" =
stored=3D"false" multiValued=3D"true"/>
        <!-- Here, default is used to create a "timestamp" field =
indicating
           When each document was indexed.
        -->
        <field name=3D"timestamp" type=3D"date" indexed=3D"true" =
stored=3D"true" default=3D"NOW" multiValued=3D"false"/>

I also used the EnwikiDocMaker from Lucene's contrib/benchmark plus a =
simple SolrJ wrapper.=