Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of srowen@gmail.com designates
 209.85.215.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CB794771-F075-4D57-94E1-1C15D5DBDF88@heystaks.com>
References: <F4ACBD96-ABF3-4E3D-922C-32163AD27910@heystaks.com>
	<CAJwFCa18VuTGAu54QdTiaqhZcBjCgeFEDk=q0VB4hHK1yYDYWw@mail.gmail.com>
	<F74C5FA5-F1D9-43A6-9AD9-CDC2032ACF4A@heystaks.com>
	<CAJwFCa14P3v18OJTk_Ox8PryNyctbGFhwXwvggkbyR8bT5_mVQ@mail.gmail.com>
	<CB794771-F075-4D57-94E1-1C15D5DBDF88@heystaks.com>
Date: Wed, 27 Feb 2013 16:05:13 +0000
Message-ID: 
 <CAEccTyxN6oCyeDA3Sbh8mjQNtEnepFti5gyPzR7H38jGW0dMpA@mail.gmail.com>
Subject: Re: Vector distance within a cluster
From: Sean Owen <srowen@gmail.com>
To: Mahout User List <user@mahout.apache.org>
Content-Type: multipart/alternative; boundary=bcaec55403ea7b298604d6b6f0f1

--bcaec55403ea7b298604d6b6f0f1
Content-Type: text/plain; charset=UTF-8

A common measure of cluster coherence is the mean distance or mean squared
difference between the members and the cluster centroid. It sounds like
this is the kind of thing you're measuring with this all-pairs distances.
That could be a measure too; I've usually seen that done by taking the
maximum such intracluster distance, the 'diameter'.

To answer Ted's question -- you're measuring internal consistency. You're
not trying to find clusters that match some external standard that says
these 100 docs should cluster together, etc.

I'm speaking off the cuff, but I think the idea was that L1/Manhattan
distance may give you clusters that tend to spread out over few rather than
more dimensions, and so that may make them more interpretable -- because
they will tend to be nearly identical in the other several dimensions and
those homogenous dimensions tell you what they're "about".

The reason is that L1 is "indifferent" across dimensions -- moving a unit
in any dimension makes you a unit further/closer from another point --
while in L2 moving along a dimension where you are already close does
little.

On Wed, Feb 27, 2013 at 3:23 PM, Chris Harrington <chris@heystaks.com>wrote:

> Hmmm, you may have to dumb things down for me here. I have don't have much
> of a background in the area of ML and I'm just piecing things together and
> learning as I go.
> So I don't really understand what you mean by "Coherence against an
> external standard?  Or internal consistency/homogeneity?" or "One thought
> along these lines is to add L_1 regularization to the k-means algorithm."
> Is L_1 regularization the same as manhattan distance?
>
> That aside I'm outputting a file with the top terms and the text of 20
> random documents that ended up in that cluster and eyeballing that, not
> very high-tech or efficient but it was the only way I knew to make a
> relevance judgment on a cluster topic. For example If the majority of the
> samples are sport related and 82.6% of the vector distances in my cluster
> are quite similar I'm happy to call that cluster sport.
>

--bcaec55403ea7b298604d6b6f0f1--