Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of clu@atypon.com designates
 74.125.245.78 as permitted sender)
Message-ID: <4E67A98B.9020101@atypon.com>
Date: Wed, 07 Sep 2011 10:27:39 -0700
From: Chris Lu <clu@atypon.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.21) Gecko/20110831 Thunderbird/3.1.13
MIME-Version: 1.0
To: user@mahout.apache.org
Subject: Re: LDA on single node is much faster than 20 nodes
References: <4E65CD3F.3040800@atypon.com>
 <CAEccTyz8pChnCzdq47VVsmSH56L7wNnCzJjdhmr9w3=wD7YTJA@mail.gmail.com>
 <CAJwFCa2qvPsag761Y3F1HNRhWjJzgReQR4QkOBD3dnKGrQ6rXQ@mail.gmail.com>
 <4E666020.4070606@atypon.com>
 <CAEccTyyzkC=bZ-btkDuX7t_y46OoKhs8F7DvBYihy6fy0rGrYQ@mail.gmail.com>
 <CAJwFCa0FPc8hT8+x0C7-gUa4SOB8HJOtxJyrSuAPx8dWqH6RZQ@mail.gmail.com>
 <4E66A54D.1020005@atypon.com>
 <CAFk0LrfhkiSs+WgriMfupxrBGfod2JtqupAXEgMBYBzJmw1fsA@mail.gmail.com>
 <CACYXym-HDJO6b2K_K9RVe-YkRiMkMyWKvVLrTQvww6g1=sxhrw@mail.gmail.com>
 <4E66B075.1020507@atypon.com>
 <CACYXym-JXVvaw5uh5_MOFwELq2G7n84vfB5NNO+z2vWJKk8nAQ@mail.gmail.com>
In-Reply-To: 
 <CACYXym-JXVvaw5uh5_MOFwELq2G7n84vfB5NNO+z2vWJKk8nAQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------030202000403020308040707"

--------------030202000403020308040707
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Thanks for the suggestions!

I finally managed to get Hadoop parallel the mapping processes.

I changed not only the "mapred.max.split.size" setting, but also 
"dfs.block.size", because of how FileInputFormat.java compute the split 
size.

   protected long computeSplitSize(long blockSize, long minSize, long 
maxSize) {
     return Math.max(minSize, Math.min(maxSize, blockSize));
   }

Now seems all nodes are running in parallel!

Chris


On 09/06/2011 04:44 PM, Jake Mannix wrote:
> On Tue, Sep 6, 2011 at 4:44 PM, Chris Lu<clu@atypon.com>  wrote:
>
>> I see, thanks!
>>
>> Seems it should build into Mahout LDA algorithms, since the input file is
>> usually not too large, but really needs parallel mapping processes.
>>
>>
> If your input is not large, running a multithreaded in-memory algorithm on a
> relatively beefy box (16+ cores, enough RAM to fit your data + model + some
> spare) will be *much* faster than putting the same data on cluster,
> actually.
>
>    -jake
>


--------------030202000403020308040707--