storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rudraneel chakraborty <rudraneel.chakrabo...@gmail.com>
Subject Re: Need Help regarding topology with numWorker>1
Date Sat, 13 Feb 2016 18:41:35 GMT
Thank you
And i need this because i am developing a custom scheduler . And for this i
need to have topologies performing well with two worker processes instead
of one. And its becoming tough to saturate a single worker

On Saturday, 13 February 2016, Matthias J. Sax <mjsax@apache.org> wrote:

> If your workload does not saturate a single machine, it will of course
> be more efficient to run within a single worker as you avoid intra-JVM
> communication.
>
> As long as CPU, memory, or network of a single machine is not utilized
> completely, you will not benefit from multiple workers from a
> performance point of view.
>
> Btw: I am a not sure what you mean by "I have to create a scenario..."?
>
> On 02/12/2016 11:42 PM, Rudraneel chakraborty wrote:
> > Thanks. Actually i have to create a scenario where 2 worker performs
> > better than one worker. But in reality , topology with a single worker
> > performs considerably better.
> >
> > I sending csv lines to kafka (5 partitions ) and reading them from a
> > topology with kafka spout (parallelism hint 5)
> >
> > Any thoughts?
> >
> > On Thursday, 11 February 2016, Nathan Leung <ncleung@gmail.com
> <javascript:;>
> > <mailto:ncleung@gmail.com <javascript:;>>> wrote:
> >
> >     Any situation where you require more CPU than 1 server can provide
> >     for you - there are tuning parameters (e.g. localOrShuffleGrouping)
> >     that you can use to reduce the amount of data sent over the network
> too.
> >
> >     Any situation where you need to have tolerance in case of machine
> >     failure.
> >
> >
> >
> >     On Thu, Feb 11, 2016 at 2:58 PM, Rudraneel chakraborty
> >     <rudraneel.chakraborty@gmail.com <javascript:;>
> >     <javascript:_e(%7B%7D,'cvml','rudraneel.chakraborty@gmail.com
> <javascript:;>');>>
> >     wrote:
> >
> >         Topology param numWorkers i meant
> >
> >
> >         On Thursday, 11 February 2016, Rudraneel chakraborty
> >         <rudraneel.chakraborty@gmail.com <javascript:;>
> >         <javascript:_e(%7B%7D,'cvml','rudraneel.chakraborty@gmail.com
> <javascript:;>');>>
> >         wrote:
> >
> >             More specifically , i have seen a topology performs better
> >             if it is assigned a single worker compared to more than one
> >             worker.
> >
> >             I want a situation where a topology performs better with
> >             more than one worker.
> >
> >             And it doesnt matter if both workers are on same supervisor
> >             or different supervisor
> >
> >             On Thursday, 11 February 2016, Matthias J. Sax
> >             <mjsax@apache.org <javascript:;>> wrote:
> >
> >                 I am not sure what you mean:
> >
> >                   - number of worker slots per supervisor
> >                 or
> >                   - topology parameter "number of workers"
> >
> >                 Can you clarify?
> >
> >                 -Matthias
> >
> >                 On 02/11/2016 05:14 AM, anshu shukla wrote:
> >                 > Not  like that..  But i have used workers equal to
> >                 number of cores. Each
> >                 > vm with 8 corea.
> >                 >
> >                 > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
> >                 > <rudraneel.chakraborty@gmail.com <javascript:;>
> >                 > <mailto:rudraneel.chakraborty@gmail.com <javascript:;>>>
> wrote:
> >                 >
> >                 >     more than one worker on same node ? Did u use
> >                 custom schedule r?
> >                 >     because by default, the workers would be spread
> >                 throughout the cluster
> >                 >
> >
> >
> >
> >             --
> >             Rudraneel Chakraborty
> >             Carleton University Real Time and Distributed Systems
> Reserach
> >
> >
> >
> >         --
> >         Rudraneel Chakraborty
> >         Carleton University Real Time and Distributed Systems Reserach
> >
> >
> >
> >
> > --
> > Rudraneel Chakraborty
> > Carleton University Real Time and Distributed Systems Reserach
> >
>
>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Mime
View raw message