Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of andyparsons@gmail.com
 designates 209.85.212.42 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=YidjsLRdjrEN3x+IKxbBC1vvDCAwsvL88U3YZ7H7903mocCTetdwnKryTLYtQr/zVt
         RGATMzNxVmrHXqsiy+1/j0DaVVkulBA1ta/X+Qw8euSWpEcU3Iy0T5URODUAq5EQXRt7
         U+8VJ1v33aifq2pfujEwtuUgarNSH9L2qFGkk=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Evaluating Mahout's recommender support
From: Andy Parsons <andyparsons@gmail.com>
In-Reply-To: <AANLkTimHGNqbboffBNnDo51eRjNyO6TwL=kFmNdd_n8q@mail.gmail.com>
Date: Wed, 29 Dec 2010 12:49:05 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <F0E759A7-D9A8-4BF8-B89E-4CD370B969D4@gmail.com>
References: <4D1AFAAA.6090206@apache.org>
 <AANLkTimHGNqbboffBNnDo51eRjNyO6TwL=kFmNdd_n8q@mail.gmail.com>
To: user@mahout.apache.org

Thanks Sean and Sebastian. I've responded to the questions inline:

On Dec 29, 2010, at 6:26 AM, Sean Owen wrote:

> Yeah that review was, IMHO, had issues. It's important to note the
> context: the person was selling their own services. It was trying to
> run some sample code, non-distributed code, in a sort of distributed
> fashion. The result was predictably not so good. That was a long time
> ago.
>=20
> 2M users and 10M items isn't big even for a non-distributed
> recommender. This doesn't even sound hard for a non-distributed Mahout
> recommender. Sure, let's hear more and we can give some ideas.
>=20
> On Wed, Dec 29, 2010 at 4:08 AM, Sebastian Schelter <ssc@apache.org> =
wrote:
>> Hi all,
>>=20
>> once again, I'm moving a twitter conversation to this mailing list.
>>=20
>> Let me introduce Andy, who is currently evaluating recommendation
>> components for his NYC located startup and looking into Mahout for =
that
>> reason:
>>=20
>> "We are coding primarily in Scala and looking to build or license a
>> recommendation component. The base requirement is that it be capable =
of
>> hybrid recommendations on a body of ~2MM users and ~10MM items with =
rich
>> metadata.  The paper I referenced seems to indicate Mahout is not a
>> great fit- can you point me to recent improvements that make the
>> assertions in the paper obsolete? Any guidance is very much =
appreciated!"
>>=20
>> The paper which he's quoting is an old review of Mahout's recommender
>> support available at
>> http://www.iletken-project.com/documents/mahout_review_by_iletken.pdf =
.
>> I think we should give great advice to Andy and simulatenously give =
the
>> community an update about the criticized facts in that review that =
are
>> not true anymore.
>>=20
>> I'll make a first try to address the state of that review:
>>=20
>>  - Mahout currently offers parallel algorithms for Collaborative
>> Filtering, see
>> =
https://cwiki.apache.org/confluence/display/MAHOUT/Itembased+Collaborative=
+Filtering
>> which can also be used to precompute a model which can than be used =
for
>> online recommendations.
>>=20
>>  - Mahout has some support for matrix factorization based =
recommenders (
>> =
https://hudson.apache.org/hudson/job/Mahout-Quality/javadoc/org/apache/mah=
out/cf/taste/impl/recommender/svd/SVDRecommender.html
>> ), a superior algrithm to this (
>> https://issues.apache.org/jira/browse/MAHOUT-525 ) as well as a =
parallel
>> implementation ( https://issues.apache.org/jira/browse/MAHOUT-542 ) =
are
>> currently in the making
>>=20
>>  -The memory consumption of Taste has significantly improved, I never
>> tried to load the Netflix dataset, but I'm pretty sure it fits into =
some
>> hundred megabytes of memory.
>>=20
>> Furthermore I think we need to know more details about Andy's usecase =
to
>> give him proper answers about Mahout fitting his project:
>>=20
>> - Do you have explicit ratings from the users or are you working with
>> implicit data?
[ASP] We will have both, in the form of ratings, views/purchases, and =
"recommend to a friend"
>>=20
>> - What do you exactly mean by hybrid recommendations? Do you mean a
>> combination of content based and collaborative filtering techniques?
[ASP] Yes, precisely.
>>=20
>> - How fast do you need the recommendations? Would it be ok to have =
them
>> precomputed on a daily basis e.g. or do you need them in realtime?
[ASP] Either *could* work, with a preference for realtime.
>>=20
>> - How often do new users and new items enter your dataset? How sparse =
is
>> your rating data?
[ASP] New users are added in the hundreds on a daily basis. Rating data =
will be very sparse in the initial months the application is live, so we =
are looking at options for priming the system. Given the quantity of =
items, however, we'll have fairly sparse rating/item coverage in =
general.
>>=20
>> --sebastian
>>=20