storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Neilson <arsneil...@gmail.com>
Subject Re: Discrepancy in Storm while Job Distribution on available nodes
Date Thu, 26 Feb 2015 07:54:37 GMT
There is a configuration setting that you should be submitting to your
topology, "topology.workers". You can see how many workers each topology is
configured to use by running "storm list" under the "num workers" column.
To confirm what Nathan suspects, you might see that this is set to 1 for
your topology. You can also see how many workers your topology is
configured to use from the Storm UI if you navigate to your running
topology and just search the page for "topology.workers".

Changing the number of workers can be done through the configuration that
you pass to the topology like (in Java):

Config conf = new Config();
conf.setNumWorkers(2);

By the way, you can find this info in the Storm tutorial:
http://storm.apache.org/documentation/Tutorial.html. There is also some
great documentation on understanding parallelism in Storm here:
http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html

On Wed, Feb 25, 2015 at 9:39 PM, Vineet Mishra <clearmidoubt@gmail.com>
wrote:

> Hi Nathan,
>
> I guess you missed the first mail of this thread.
>
> As mentioned before I m having 3 node cluster out of which node 1 is for
> nimbus and ui while other 2 nodes are for worker and supervisor.
>
> The specification of machine is 4 core intel xeon processor which
> currently is running with more than 300% cpu utilization.
>
> At this case how can I Increase number of workers
>
> Thanks!
> On Feb 26, 2015 3:48 AM, "Nathan Leung" <ncleung@gmail.com> wrote:
>
>> If you have only one node on the cluster then yes I would recommend to
>> add more nodes. If you have more than one I would recommend you increase
>> the worker processes in the topology configuration to use the other
>> hardware.
>> On Feb 25, 2015 10:38 AM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:
>>
>>> If I am taking you right, by adding more workers you mean adding more
>>> nodes to the existing cluster or/and enhancing the existing configuration
>>> with perhaps 8 or 16 core cpu?
>>>
>>> On Wed, Feb 25, 2015 at 6:59 PM, Nathan Leung <ncleung@gmail.com> wrote:
>>>
>>>> In practice it's very difficult to fully maximize cpu usage, especially
>>>> when each thread has to do things like wait for i/o in any way (network,
>>>> disk, etc).  Since you're cpu limited, or very close to it, you should try
>>>> adding more workers.
>>>> On Feb 25, 2015 7:48 AM, "Vineet Mishra" <clearmidoubt@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Nathan,
>>>>>
>>>>> You are right, I guess that was the case.
>>>>>
>>>>> But I am not sure why even after increasing the spouts to 20 I
>>>>> couldn't end up with max throughput, I hold 4 core Xeon processor Intel(R)
>>>>> Xeon(R) CPU E5-2680 v2 @ 2.80GHz, although I could see the high CPU
>>>>> utilization reaching more than 350% but is it the only reason behind
the
>>>>> less performance gain than expected or do I need to add some configuration
>>>>> tuning.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Wed, Feb 25, 2015 at 5:53 PM, Nathan Leung <ncleung@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> It looks like you configured your topology to only use one worker
>>>>>> process.
>>>>>>
>>>>>> On Wed, Feb 25, 2015 at 6:39 AM, Vineet Mishra <
>>>>>> clearmidoubt@gmail.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I am running a 3 node storm cluster with a Topology which is
>>>>>>> subscribed to Kafka using Kafka-Storm Spout.
>>>>>>>
>>>>>>> The Cluster is hosting Nimbus on a node and left with 2 other
nodes
>>>>>>> for the workers and supervisor to function.
>>>>>>>
>>>>>>> While running the Topology I could see that all the spouts(20)
and
>>>>>>> bolts(5) tasks are running on the same machine leaving the other
node
>>>>>>> instance without any CPU utilization.
>>>>>>>
>>>>>>>   [1-1] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2264180
2264180
>>>>>>> 11.519 2262620 0  [2-2] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 841540
841540
>>>>>>> 19.717 840940 0  [3-3] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1080080
1080080
>>>>>>> 18.105 1079240 0  [4-4] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1113900
1113900
>>>>>>> 17.966 1105320 0  [5-5] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854220
854220
>>>>>>> 19.342 856240 0  [6-6] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1249640
1249640
>>>>>>> 17.071 1251320 0  [7-7] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2266520
2266520
>>>>>>> 11.63 2267260 0  [8-8] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 629760
629760
>>>>>>> 20.8 629120 0  [9-9] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854360
854360
>>>>>>> 19.513 851800 0  [10-10] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1417380
1417380
>>>>>>> 15.78 1418200 0  [11-11] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 858620
858620
>>>>>>> 20.897 858180 0  [12-12] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1242980
1242980
>>>>>>> 16.766 1241100 0  [13-13] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 352300
352300
>>>>>>> 19.975 349420 0  [14-14] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 627920
627920
>>>>>>> 20.329 625440 0  [15-15] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1061880
1061880
>>>>>>> 18.263 1061300 0  [16-16] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 706480
706480
>>>>>>> 20.297 702720 0  [17-17] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1518800
1518800
>>>>>>> 15.154 1514640 0  [18-18] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 622640
622640
>>>>>>> 20.743 626240 0  [19-19] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 3376960
3376960
>>>>>>> 8.361 3377060 0  [20-20] 31m 7s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1044940
1044940
>>>>>>> 18.569 1043760 0
>>>>>>>
>>>>>>> Same is the case for bolts,
>>>>>>>
>>>>>>>   [22-22]} 32m 13s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0
0.002 0.05
>>>>>>> 1376560 0.042 1376540 0  [23-23]} 32m 13s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0
0.001 0.021
>>>>>>> 347180 0.02 347180 0  [24-24]} 32m 13s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0
0 0.023
>>>>>>> 34600 0.023 34600 0  [25-25]} 32m 13s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0
0 0.017 8380
>>>>>>> 0.014 8380 0  [26-26]} 32m 13s ip-20-0-0-75 6703
>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0
0.082 0.039
>>>>>>> 22217940 0.041 22217920 0
>>>>>>>
>>>>>>> ip-20-0-0-75 is only utilized while their is another host which
is
>>>>>>> not utilized at all. Please find the supervisor summary below
>>>>>>>
>>>>>>>   8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h 7m
32s 4 0
>>>>>>> 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h 57m
53s 4 1
>>>>>>> This is not the regular behavior of Storm as I have seen both
the
>>>>>>> nodes operating while running the jobs, but how we can comment
on the
>>>>>>> stability of the storm with this kind off behavior.
>>>>>>>
>>>>>>> Urgent Issue, Request for expert intervention.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>

Mime
View raw message