Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of srowen@gmail.com designates
 209.85.214.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAC-eW67s2dek0okkn8u2qfKo7br=+ChacaZxtcGuULrVCELOJQ@mail.gmail.com>
References: 
 <CAC-eW67jbZvUwd9kjwK-RG=MUDiJTzZ4FC4yQp_sBGj5Kk7CKQ@mail.gmail.com>
	<CAC-eW66v9t3tW-7byEJiF0AaPL0pvM028ugiAoY+_eCUUbvsxQ@mail.gmail.com>
	<CAEccTyyW-f+F1fNpyUDSQvcCdB2msuHKuwt67NTit7uxdkkCDw@mail.gmail.com>
	<CAC-eW66BYTpFwZ=ggtXOvy+yDs+i+KM8dLUX_VNCkmZncj8h=g@mail.gmail.com>
	<CAC-eW64xNi2f_W-eJ4vSFqGo6v3D3o=+3POoZrw7v93FPNq0sA@mail.gmail.com>
	<CAEccTywm3o59TQGphO5EDBL4uVWe5XkErbpr_U77K2BcQJ-3Tw@mail.gmail.com>
	<CA+sfuFS9SrEW+yeKtHK2CAy1Lseds84sVhm_3pwmXyLOMeho4g@mail.gmail.com>
	<CAEccTyy0_A-Sxb8pbsFmcG=L9Ahj8-K1Y6wfd3c2hotZHzLtqg@mail.gmail.com>
	<CADHDM+Yofc1PLOA=jx0v8FcwoS1kkyB0=kPzPzA9tqhBEuekgg@mail.gmail.com>
	<EBAB5770-F560-4CE0-AEA3-828A206CEDF5@gmx.de>
	<CAC-eW67supvt2Rd4AWoeKYcPL-JPt4t+bt+5m6Lf1ONDskOKwQ@mail.gmail.com>
	<21068ABA-D5DD-4AB0-9BCF-178D4688E120@gmx.de>
	<CAC-eW64eUO0NKU8gij-jwku1zS64UZ6hegz2BzN8QsheoQ7OJw@mail.gmail.com>
	<CAEccTyxAFQhnRU+VqRJ+v4NZa7A+F+9o068d+5w6r4GFAEsrYg@mail.gmail.com>
	<4ED79A4D.40209@apache.org>
	<4ED7C32A.6090504@apache.org>
	<CAC-eW677MMbAHE-4CyYb3+uxObDvngoKHEPhDXpvvoxDjirfHA@mail.gmail.com>
	<CAJwFCa0V9ehSmPg=tTBrziFyOy+9aWPmvFRkjA8OyZ6XAVR5HA@mail.gmail.com>
	<CAEccTyxVx5nc7NZeJT5GU7omDoX3Scf9jbSixv3NnDpzHCNneQ@mail.gmail.com>
	<CAC-eW661wvW6Nv6hGb+TU8-Mj0var2SnOt9ie=mReUcHgHvYXQ@mail.gmail.com>
	<CAEccTyyiKOCFookW3bTUrfUO1WvpELKOuMryZcgW_VX4HckXzg@mail.gmail.com>
	<CAC-eW64HbaYjf++3uLFPUfBUNbW3E35d5EsN0A4ZXg=+fr+itw@mail.gmail.com>
	<CAEccTyyLwTHiOajqZ7Y5uc4iRCeTruDfO4vMOZRRA1QoWYrqGg@mail.gmail.com>
	<CAC-eW65OrkXJ6MawEKfg9dZGjF08GqArDot_coHocpX0Q5mNwQ@mail.gmail.com>
	<CAEccTywkoiDeT-g+uTf2p99BCuX8agAP8AuWk6+4BJ41nhL_VQ@mail.gmail.com>
	<CAC-eW65TW_FfoFR7Rryk5x=V70+DigL+Le4Fdr3OkFXKMdWnmg@mail.gmail.com>
	<CAEccTyzUPd+C8Z_6+YZ9651+=ybF2CPpGeQEQw8STOb0MNVOVQ@mail.gmail.com>
	<CAC-eW67s2dek0okkn8u2qfKo7br=+ChacaZxtcGuULrVCELOJQ@mail.gmail.com>
Date: Fri, 2 Dec 2011 18:19:44 +0000
Message-ID: 
 <CAEccTywK+xtFU0RDHB_pZfyERWT68JghaMhx4rrVQZPT4Z2Qog@mail.gmail.com>
Subject: Re: Mahout performance issues
From: Sean Owen <srowen@gmail.com>
To: user@mahout.apache.org
Content-Type: multipart/alternative; boundary=0015175cfdee72f36304b3200362

--0015175cfdee72f36304b3200362
Content-Type: text/plain; charset=UTF-8

On Fri, Dec 2, 2011 at 6:07 PM, Daniel Zohar <dissoman@gmail.com> wrote:

> > I definitely agree that the correctness should not be broken. My solution
> is not meant to decrease the number of possible items like you stated in
> your example. It was meant to reduce the amount of item-user associations
> (while preserving user-item associations) which will results much less
> effort on intersectionSize(). Even in the case that we have two popular
>

My point is that intersectionSize() is called as part of a similarity
computation. Yes, that's the bottleneck. But, that happens after the stage
where candidate items are identified. And you are talking about changing
the candidate identification stage, which is not the bottleneck.

I think your change *happens* to also reduce the number of similarity
computations since it assumes some are 0, when they are not! sure that
saves time, in the same way that you'll finish an exam faster if you don't
answer half the questions.

I am instead suggesting to optimize intersectionSize(), such that for all
of these 1-item cases, the answer is computed extremely fast. Which also
addresses the bottleneck of course.


I suppose this could be proven or disproven quickly -- do you get the same
speed up with the change I committed, without your change? if you do,
great, we have a solution. If not then I am wrong and you have some example
that pinpoints where the new bottleneck is.

--0015175cfdee72f36304b3200362--