Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of srowen@gmail.com designates
 74.125.83.42 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=aM0smNQLnZeRWrYhIEPKoZwiY2MBpQlZKA85RXkakZVmuITDkzr8WEBg/VQaXXGHgO
         7xA7nAFnaPZUd44EaSXRDqHFKMCJmg7dCv38BE6EYtFUeIpNNgs5zycMHTXwZiTIZHL1
         UjdhLJ2AnlVvDHlh0PShfziPqudfsxgsO1Utc=
MIME-Version: 1.0
In-Reply-To: <AANLkTikubZmo7ojNYfz99HFLdiY=s-nMZCHjeJitVvuq@mail.gmail.com>
References: <AANLkTikubZmo7ojNYfz99HFLdiY=s-nMZCHjeJitVvuq@mail.gmail.com>
Date: Fri, 19 Nov 2010 22:04:01 +0000
Message-ID: <AANLkTimnZOy2P0izGqWQ0f2gfMK7PQyvN0ZTqqNG5eUD@mail.gmail.com>
Subject: Re: Need for a distributed SVDRecommender
From: Sean Owen <srowen@gmail.com>
To: user@mahout.apache.org
Content-Type: multipart/alternative; boundary=00032557a0d27b34f704956f1522

--00032557a0d27b34f704956f1522
Content-Type: text/plain; charset=UTF-8

That result sounds confusing. It should take about the same number of
wall-clock hours either way. I don't see why it would take 14 hours -- that
sounds wrong. If anything it should take 38 / N minutes where N is the
number of recommenders
you ran.

SVDRecommender is not distributed at all, no.

On Fri, Nov 19, 2010 at 9:34 PM, Sanjib Kumar Das <sanjib.kgp@gmail.com>wrote:

> Hi All,
>
> I wanted to run a distributed RecommenderJob with the SVDRecommender
> implementation.
> So i ran the pseudo.RecommenderJob with an
> SVDRecommender(numFeatures=30,trainingSteps=50) on the 1M Movielens
> data(6040 users). So this generated 10 recommendations for each of the 6040
> users but took 14 hours to do so! My hadoop cluster had 12 m/cs. So i guess
> it just ran multiple instances of the non-distributed SVD implementation
> and
> each of these instances did the same thing again and again. So unless the
> implementation of the recommender is distributed, we dont get any special
> benefit with the pseudo.RecommenderJob.
>
> But the item.RecommenderJob does the same 10 recommendations each for the
> 6040 users in 38 minutes. This is because it has an underlying distributed
> implementation.
>
> So my doubt is do we have a distributed SVDRecommender implementation? If
> not, how should i go about writing one? Can I use the new LanczosSolver to
> achieve this?
>
> Thanks,
> Sanjib
>

--00032557a0d27b34f704956f1522--