mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] Resolved: (MAHOUT-372) Partitioning Collaborative Filtering Job into Maps and Reduces
Date Fri, 09 Apr 2010 12:27:50 GMT


Sean Owen resolved MAHOUT-372.

       Resolution: Fixed
    Fix Version/s: 0.4
         Assignee: Sean Owen

Yes, sure there's no particular limit to the number of mappers or reducers. 

These are Hadoop params, which you can set on the command line with, for example: -Dmapred.reduce.tasks=10

Reopen if that doesn't quite answer the question. (We can also discuss on,
perhaps, if this isn't necessarily a bug or enhancement request.)

> Partitioning Collaborative Filtering Job into Maps and Reduces
> --------------------------------------------------------------
>                 Key: MAHOUT-372
>                 URL:
>             Project: Mahout
>          Issue Type: Question
>          Components: Collaborative Filtering
>    Affects Versions: 0.4
>         Environment: Ubuntu Koala
>            Reporter: Kris Jack
>            Assignee: Sean Owen
>             Fix For: 0.4
> I am running the main on my hadoop
cluster and it partitions the job in 2 although I have more than 2 nodes available.  I was
reading that the partitioning could be changed by setting the JobConf's conf.setNumMapTasks(int
num) and conf.setNumReduceTasks(int num).
> Would I be right in assuming that this would speed up the processing by increasing these,
say to 4)?  Can this code be partitioned into many reducers?  If so, would setting them in
the protected AbstractJob::JobConf prepareJobConf() function be appropriate?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message