apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junguk Cho <jman...@gmail.com>
Subject Re: Dynamic Application Modification
Date Fri, 10 Jun 2016 21:39:41 GMT
Hi.

Thank you for replies.

In Storm, storm APIs provides several grouping methods(
http://storm.apache.org/releases/current/Concepts.html).
For example, app developers specify the number of replica and routing rules
like shuffleGrouping and fieldsGrouping.

builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new
Fields("word"));

In apex, we can set the number of replica in property files and for
partitions except for "sticky" we can use "Partition" and StreamCodec.

Is my understanding correct? Thanks in advance. Junguk




2016-06-10 16:16 GMT-04:00 Munagala Ramanath <ram@datatorrent.com>:

> There is no mechanism for automatic load balancing across the board; some
> individual operators
> support it (e.g. Kafka input operator) out of the box. You can implement
> it in your own operator by using a
> custom partitioner.
>
> Keys are sticky by default since the hashcode is used to distribute tuples
> to partitions. There is
> no special setting for this. You can gain more control over how the
> distribution happens by
> using a StreamCodec.
>
> Ram
>
> On Fri, Jun 10, 2016 at 11:36 AM, Junguk Cho <jmanbal@gmail.com> wrote:
>
>> Hi, Ram.
>>
>> Thank you for reply.
>> I have following question.
>>
>> 1. Where do we specify policy to use "Load balance" or "Sticky Key"?
>>
>> 2. This is just general question.
>> For example, I would like to run A class now as one operator and then
>> later I need other tasks which I do not think.
>> At that time, could I load this new class in current DAG even though it
>> is not included in *.apa file.
>>
>>
>> Thanks,
>> Junguk
>>
>>
>> 2016-06-10 12:03 GMT-04:00 Munagala Ramanath <ram@datatorrent.com>:
>>
>>> You don't need dynamic partitioning to achieve that topology. You can
>>> simply create your DAG as: A --> X --> Y and then set the *PARTITIONER*
>>> attribute on X
>>> as discussed in the "Advanced Features" section of the TopN words
>>> tutorial at:
>>> http://docs.datatorrent.com/tutorials/topnwords-c7/
>>>
>>> The stanza in your properties file should look something like this (with
>>> suitable changes
>>> for the application and operator names):
>>>
>>> *<property>*
>>> *
>>> <name>dt.application.TopNWordsWithQueries.operator.wordReader.attr.PARTITIONER</name>*
>>> *
>>> <value>com.datatorrent.common.partitioner.StatelessPartitioner:2</value>*
>>> *</property>*
>>>
>>> The platform will then create two replicas of X.
>>>
>>> For the rest, could you provide additional details of what you are
>>> trying to accomplish, what
>>> you tried and what sort of problems you encountered ? That will help us
>>> provide more precise answers.
>>>
>>> Ram
>>>
>>> On Fri, Jun 10, 2016 at 8:37 AM, Junguk Cho <jmanbal@gmail.com> wrote:
>>>
>>>> Hi.
>>>>
>>>> I explore "Dynamic Application Modification" with this example(
>>>> https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition
>>>> ).
>>>>
>>>>
>>>> To be clear, basically, application developers carefully design a
>>>> program.
>>>> For Dynamic Partitioning, in "dynamic partition" example, it uses
>>>> threadhold to adjust partition. In addition, to use Modification of DAG
>>>> structure, app developers should make program like java classes to use them
>>>> later.
>>>> Do I interpret that they are determined "compiling time"?
>>>>
>>>> How do I use "Load balance and Sticky Key" in Apex?
>>>> I would like to create this topology for word counting.
>>>>  For example,
>>>>                                          B operator
>>>>                      A operator ->                        -> one
>>>> onperator
>>>>                                          C Operator
>>>>
>>>> A sends the same values like "word" to the same operator ( B or C )?
>>>> Can we specify this rule in code and how to make two process for B and
>>>> C with one class?
>>>>
>>>> Thanks,
>>>> Junguk
>>>>
>>>
>>>
>>
>

Mime
View raw message