incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Vnodes - HUNDRED of MapReduce jobs
Date Fri, 29 Mar 2013 14:28:38 GMT
I still don't see the hole in the following reasoning:

- Input splits are 64k by default.  At this size, map processing time
dominates job creation.
- Therefore, if job creation time dominates, you have a toy data set
(< 64K * 256 vnodes = 16 MB)

Adding complexity to our inputformat to improve performance for this
niche does not sound like a good idea to me.

On Thu, Mar 28, 2013 at 8:40 AM, cem <cayiroglu@gmail.com> wrote:
> Hi Alicia ,
>
> Cassandra input format creates mappers as many as vnodes. It is a known
> issue. You need to lower the number of vnodes :(
>
> I have a simple solution for that and ready to write a patch. Should I
> create a ticket about that? I don't know the procedure about that.
>
>  Regards,
> Cem
>
>
> On Thu, Mar 28, 2013 at 2:30 PM, Alicia Leong <lccalicia@gmail.com> wrote:
>>
>> Hi All,
>>
>> I have 3 nodes of Cassandra 1.2.3 & edited the cassandra.yaml for vnodes.
>>
>> When I execute a M/R job .. the console showed HUNDRED of Map tasks.
>>
>> May I know, is the normal since is vnodes?  If yes, this have slow the M/R
>> job to finish/complete.
>>
>>
>> Thanks
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Mime
View raw message