storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From padma priya chitturi <padmapriy...@gmail.com>
Subject Re: Basic storm question
Date Thu, 03 Apr 2014 04:25:34 GMT
Hi,

This is how you should run nimbus/supervisor:

/bin$./storm nimbus
/bin$./storm supervisor


On Wed, Apr 2, 2014 at 11:42 PM, Leonardo Bohac <leonardo.bohac@gmail.com>wrote:

> Hello, I've downloaded the last version of storm at
> http://storm.incubator.apache.org/downloads.html and when I try to do the */bin/storm
> nimbus* command I get the following message:
>
> *The storm client can only be run from within a release. You appear to be
> trying to run the client from a checkout of Storm's source code.*
>
> *You can download a Storm release
> at http://storm-project.net/downloads.html
> <http://storm-project.net/downloads.html>*
>
>
>
>  I don't know whats missing...
>
>
> Thanks!
>
>
> 2014-04-02 15:05 GMT-03:00 Nathan Leung <ncleung@gmail.com>:
>
> No, it creates an extra executor to deal with processing the ack messages
>> that are sent by the bolts after processing tuples.  See the following for
>> details on how acking works in storm:
>> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing.
>>  By default storm will create 1 acker per worker you have in your topology.
>>
>>
>> On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang <zhlntu@gmail.com> wrote:
>>
>>> Hi Nathan,
>>>
>>> The last bolt just emits the tuples and no more bolt in the topology
>>> will consume and ack the tuples. Do you mean that storm automatically
>>> creates an extra executor to deal with the tuples?
>>>
>>> Thanks,
>>> Huiliang
>>>
>>>
>>> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung <ncleung@gmail.com> wrote:
>>>
>>>> the extra task/executor is the acker thread.
>>>>
>>>>
>>>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang <zhlntu@gmail.com>wrote:
>>>>
>>>>> I just submitted ExclamationTopology for testing.
>>>>>
>>>>>     builder.setSpout("word", new TestWordSpout(), 10);
>>>>>
>>>>>     builder.setBolt("exclaim1", new ExclamationBolt(),
>>>>> 3).shuffleGrouping("word");
>>>>>
>>>>>     builder.setBolt("exclaim2", new ExclamationBolt(),
>>>>> 2).shuffleGrouping("exclaim1");
>>>>>
>>>>> I am supposed to see 15 executors. However, I see 16 executors and 16
>>>>> tasks on topology summary on storm UI. The numbers of executors are correct
>>>>> for the specific spout and bolts and aggregate to 15. Is that a bug in
>>>>> displaying topology summary?
>>>>>
>>>>> My cluster consists of 2 supervisors and each has 4 workers defined.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung <ncleung@gmail.com>wrote:
>>>>>
>>>>>> By default supervisor nodes can run up to 4 workers.  This is
>>>>>> configurable in storm.yaml (for example see supervisor.slots.ports
here:
>>>>>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
>>>>>>  Memory should be split between the workers.  It's a typical Java
heap, so
>>>>>> anything running on that worker process shares the heap.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland <
>>>>>> david@elastacloud.com> wrote:
>>>>>>
>>>>>>>  On said subject, how does memory allocation work I these cases?
>>>>>>> Assuming 1 worker per node would you just dump all the memory
available
>>>>>>> into worker.childopts? I guess the memory pool would be shared
between the
>>>>>>> spawned threads as appropriate to their needs?
>>>>>>>
>>>>>>>  I'm assuming the equivalent options for supervisor/nimbus are
fine
>>>>>>> left at defaults.  Given that the workers/spouts/bolts are the
working
>>>>>>> parts of the topology these would where I should target available
memory?
>>>>>>>
>>>>>>>  D
>>>>>>>
>>>>>>>   *From:* Huiliang Zhang <zhlntu@gmail.com>
>>>>>>> *Sent:* Tuesday, 1 April 2014 19:47
>>>>>>> *To:* user@storm.incubator.apache.org
>>>>>>>
>>>>>>>  Thanks. It should be good if there exist some example figures
>>>>>>> explaining the relationship between tasks, workers, and threads.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
>>>>>>> skgadalay@gmail.com> wrote:
>>>>>>>
>>>>>>>> No, a single worker is dedicated to a single topology no
matter how
>>>>>>>> many threads it spawns for different bolts/spouts.
>>>>>>>> A single worker cannot be shared across multiple topologies.
>>>>>>>>
>>>>>>>> On 3/29/14, Nathan Leung <ncleung@gmail.com> wrote:
>>>>>>>> > From what I have seen, the second topology is run with
1 worker
>>>>>>>> until you
>>>>>>>> > kill the first topology or add more worker slots to
your cluster.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang <zhlntu@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> >> Thanks. I am still not clear.
>>>>>>>> >>
>>>>>>>> >> Do you mean that in a single worker process, there
will be
>>>>>>>> multiple
>>>>>>>> >> threads and each thread will handle part of a topology?
If so,
>>>>>>>> what does
>>>>>>>> >> the number of workers mean when submitting topology?
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi
<
>>>>>>>> >> padmapriya30@gmail.com> wrote:
>>>>>>>> >>
>>>>>>>> >>> Hi,
>>>>>>>> >>>
>>>>>>>> >>> No, its not the case. No matter how many topologies
you submit,
>>>>>>>> the
>>>>>>>> >>> workers will be shared among the topologies.
>>>>>>>> >>>
>>>>>>>> >>> Thanks,
>>>>>>>> >>> Padma Ch
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang
<
>>>>>>>> zhlntu@gmail.com>
>>>>>>>> >>> wrote:
>>>>>>>> >>>
>>>>>>>> >>>> Hi,
>>>>>>>> >>>>
>>>>>>>> >>>> I have a simple question about storm.
>>>>>>>> >>>>
>>>>>>>> >>>> My cluster has just 1 supervisor and 4 ports
are defined to
>>>>>>>> run 4
>>>>>>>> >>>> workers. I first submit a topology which
needs 3 workers. Then
>>>>>>>> I submit
>>>>>>>> >>>> another topology which needs 2 workers.
Does this mean that
>>>>>>>> the 2nd
>>>>>>>> >>>> topology will never be run?
>>>>>>>> >>>>
>>>>>>>> >>>> Thanks,
>>>>>>>> >>>> Huiliang
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message