storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Need Help regarding topology with numWorker>1
Date Sat, 13 Feb 2016 16:45:29 GMT
If your workload does not saturate a single machine, it will of course
be more efficient to run within a single worker as you avoid intra-JVM
communication.

As long as CPU, memory, or network of a single machine is not utilized
completely, you will not benefit from multiple workers from a
performance point of view.

Btw: I am a not sure what you mean by "I have to create a scenario..."?

On 02/12/2016 11:42 PM, Rudraneel chakraborty wrote:
> Thanks. Actually i have to create a scenario where 2 worker performs
> better than one worker. But in reality , topology with a single worker
> performs considerably better. 
> 
> I sending csv lines to kafka (5 partitions ) and reading them from a
> topology with kafka spout (parallelism hint 5)
> 
> Any thoughts? 
> 
> On Thursday, 11 February 2016, Nathan Leung <ncleung@gmail.com
> <mailto:ncleung@gmail.com>> wrote:
> 
>     Any situation where you require more CPU than 1 server can provide
>     for you - there are tuning parameters (e.g. localOrShuffleGrouping)
>     that you can use to reduce the amount of data sent over the network too.
> 
>     Any situation where you need to have tolerance in case of machine
>     failure.
> 
> 
> 
>     On Thu, Feb 11, 2016 at 2:58 PM, Rudraneel chakraborty
>     <rudraneel.chakraborty@gmail.com
>     <javascript:_e(%7B%7D,'cvml','rudraneel.chakraborty@gmail.com');>>
>     wrote:
> 
>         Topology param numWorkers i meant 
> 
> 
>         On Thursday, 11 February 2016, Rudraneel chakraborty
>         <rudraneel.chakraborty@gmail.com
>         <javascript:_e(%7B%7D,'cvml','rudraneel.chakraborty@gmail.com');>>
>         wrote:
> 
>             More specifically , i have seen a topology performs better
>             if it is assigned a single worker compared to more than one
>             worker. 
> 
>             I want a situation where a topology performs better with
>             more than one worker.
> 
>             And it doesnt matter if both workers are on same supervisor
>             or different supervisor
> 
>             On Thursday, 11 February 2016, Matthias J. Sax
>             <mjsax@apache.org> wrote:
> 
>                 I am not sure what you mean:
> 
>                   - number of worker slots per supervisor
>                 or
>                   - topology parameter "number of workers"
> 
>                 Can you clarify?
> 
>                 -Matthias
> 
>                 On 02/11/2016 05:14 AM, anshu shukla wrote:
>                 > Not  like that..  But i have used workers equal to
>                 number of cores. Each
>                 > vm with 8 corea.
>                 >
>                 > On 11 Feb 2016 9:07 am, "Rudraneel chakraborty"
>                 > <rudraneel.chakraborty@gmail.com
>                 > <mailto:rudraneel.chakraborty@gmail.com>> wrote:
>                 >
>                 >     more than one worker on same node ? Did u use
>                 custom schedule r?
>                 >     because by default, the workers would be spread
>                 throughout the cluster
>                 >
> 
> 
> 
>             -- 
>             Rudraneel Chakraborty
>             Carleton University Real Time and Distributed Systems Reserach
> 
> 
> 
>         -- 
>         Rudraneel Chakraborty
>         Carleton University Real Time and Distributed Systems Reserach
> 
> 
> 
> 
> -- 
> Rudraneel Chakraborty
> Carleton University Real Time and Distributed Systems Reserach
> 


Mime
View raw message