incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Vnodes - HUNDRED of MapReduce jobs
Date Fri, 29 Mar 2013 18:17:49 GMT
Yes but my point, is with 50 map slots you can only be processing 50 at
once. So it will take 1000/50 "waves" of mappers to complete the job.


On Fri, Mar 29, 2013 at 11:46 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> My point is that if you have over 16MB of data per node, you're going
> to get thousands of map tasks (that is: hundreds per node) with or
> without vnodes.
>
> On Fri, Mar 29, 2013 at 9:42 AM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
> > Every map reduce task typically has a minimum Xmx of 256MB memory. See
> > mapred.child.java.opts...
> > So if you have a 10 node cluster with 256 vnodes... You will need to
> spawn
> > 2,560 map tasks to complete a job.
> > And a 10 node hadoop cluster with 5 map slotes a node... You have 50 map
> > slots.
> >
> > Wouldnt it be better if the input format spawned 10 map tasks instead of
> > 2,560?
> >
> >
> > On Fri, Mar 29, 2013 at 10:28 AM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >> I still don't see the hole in the following reasoning:
> >>
> >> - Input splits are 64k by default.  At this size, map processing time
> >> dominates job creation.
> >> - Therefore, if job creation time dominates, you have a toy data set
> >> (< 64K * 256 vnodes = 16 MB)
> >>
> >> Adding complexity to our inputformat to improve performance for this
> >> niche does not sound like a good idea to me.
> >>
> >> On Thu, Mar 28, 2013 at 8:40 AM, cem <cayiroglu@gmail.com> wrote:
> >> > Hi Alicia ,
> >> >
> >> > Cassandra input format creates mappers as many as vnodes. It is a
> known
> >> > issue. You need to lower the number of vnodes :(
> >> >
> >> > I have a simple solution for that and ready to write a patch. Should I
> >> > create a ticket about that? I don't know the procedure about that.
> >> >
> >> >  Regards,
> >> > Cem
> >> >
> >> >
> >> > On Thu, Mar 28, 2013 at 2:30 PM, Alicia Leong <lccalicia@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi All,
> >> >>
> >> >> I have 3 nodes of Cassandra 1.2.3 & edited the cassandra.yaml for
> >> >> vnodes.
> >> >>
> >> >> When I execute a M/R job .. the console showed HUNDRED of Map tasks.
> >> >>
> >> >> May I know, is the normal since is vnodes?  If yes, this have slow
> the
> >> >> M/R
> >> >> job to finish/complete.
> >> >>
> >> >>
> >> >> Thanks
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder, http://www.datastax.com
> >> @spyced
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>

Mime
View raw message