Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-Id: <8092565B-5EEC-467C-BD25-258A5E5F34C3@apache.org>
From: Grant Ingersoll <gsingers@apache.org>
To: mahout-user@lucene.apache.org
In-Reply-To: <37ffc8080907230720t4429386s208c5eb8cfce6b24@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Clustering from DB
Date: Thu, 23 Jul 2009 12:50:33 -0400
References: <37ffc8080906260720w485c1babq9b0b765c07e9e0ac@mail.gmail.com>
 <B7E154E6-7C72-4403-AA5C-93D71EAE65F2@apache.org>
 <518b17a0907151446p60e5f38n7a816b0b4c288976@mail.gmail.com>
 <c7d45fc70907151523i3d6976bbpb81a562ed647124a@mail.gmail.com>
 <37ffc8080907160539v182bd462yf1e8bdb611f8066c@mail.gmail.com>
 <37ffc8080907200930v18ee47c4le01f7eb662bbc42f@mail.gmail.com>
 <37ffc8080907201158x1688f3c1ydf7bb20205c3956d@mail.gmail.com>
 <37ffc8080907210505m6e37a0a0m333c0b1c65fbe91c@mail.gmail.com>
 <37ffc8080907220722p4ea08ee8sd39dd8d2a9738310@mail.gmail.com>
 <4CB4FAE0-6995-4ECB-ACB1-9A8673B8A032@apache.org>
 <37ffc8080907230720t4429386s208c5eb8cfce6b24@mail.gmail.com>


On Jul 23, 2009, at 10:20 AM, nfantone wrote:

>> That does seem like a long time.
>>
>> Is your data sparse or dense?
>
> I would say sparse. My vectors are high dimensional and most of their
> values are zero.
>
>> Perhaps a larger convergence value might help (-d, I believe).
>
> I'll try that.
>
>> Is there any chance your data is publicly shareable?  Come to think  
>> of it,
>> with the vector representations, as long as you don't publish the  
>> key (which
>> terms map to which index), I would think most all data is publicly
>> shareable.
>
> I'm sorry, I don't quite understand what you're asking. Publicly
> shareable? As in user-permissions to access/read/write the data?

As in post a copy of the SequenceFile somewhere for download, assuming  
you can.  Then others could presumably try it out.


>
>> Are you on trunk of Mahout?  I think we still need more profiling  
>> to get a
>> better idea of where improvements can be made.
>
> I am. Updated this morning.
>
> I still insist on the configuration issue, and have never considered
> Mahout's algorithms implementation to be the actual cause of poor
> performance. For now, I've been running kMeans exclusively. Perhaps, I
> should try with different clustering methods and see if it takes a
> similar amount of time to complete.

Well KMeans actually runs two algorithms normally: canopy and then  
KMeans.  You could try the Random seed approach, which would skip the  
canopy run first.