Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.220.41 as permitted sender)
Subject: Re: Streaming kmeans question
References: 
 <CA+18KkFj1yT0wrT94xgvyaNzhYcN7LACF0rB-qfPxCHEkghVKQ@mail.gmail.com>
 <CAJwFCa387KLodt6bpf9t_R8y70NT+9Vcf0ELL0T81HEFOhb8EA@mail.gmail.com>
 <CA+18KkEvihmt83q7Nfn7yEPu46ifE_k+79x6M2fFtnMWS+BDSg@mail.gmail.com>
From: Ted Dunning <ted.dunning@gmail.com>
Content-Type: text/plain;
	charset=utf-8
In-Reply-To: 
 <CA+18KkEvihmt83q7Nfn7yEPu46ifE_k+79x6M2fFtnMWS+BDSg@mail.gmail.com>
Message-Id: <E6DF7F7A-01DE-47A3-AA14-70B7973ABD05@gmail.com>
Date: Mon, 28 Jul 2014 13:45:34 -0600
To: "user@mahout.apache.org" <user@mahout.apache.org>
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)


I am traveling and it is difficult to get a real internet connection.=20


Here is an answer one of your questions.=20

For very dimension data, some kind of dimensionality reduction is usually im=
portant. The streaming k-means code does the by approximating the nearest ce=
ntroid by using a random projection.=20

Note that the output of the streaming step is *not* a set of initial centroi=
ds. Instead it is a large number of centroids which are clustered as a surro=
gate for the original data.  These centroids are much less numerous than the=
 original data so the final ball k-means can run in memory. This is very dif=
ferent than the canopy approach.=20

There is a known issue with the map-reduce version of the streaming k-means p=
rogram that causes the number of centroids output by the parallel part of th=
e algorithm to be too large.=20

There is a known issue


Sent from my iPhone

> On Jul 28, 2014, at 3:08, Bojan Kosti=C4=87 <blood9raven@gmail.com> wrote:=

>=20
> Also as i see this stream kmeans is for large sets of data. Does this larg=
e
> means large number of points and not dimmensions? And what to do when data=

> have large dimensions? Like more then 1000000 dimensions.