samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mhaskar, Tushar" <tmhas...@paypal.com.INVALID>
Subject Re: [DISCUSS] Sliding window implementation in Samza
Date Wed, 26 Aug 2015 16:39:20 GMT
Thanks for the response but I have one more question .

Lets say I am consuming from a Kafka topic which has 5 partitions and I am
implementing the WindowableTask interface .
There will be 5 Task instances created to consume data from 5 partitions.
The window function will also be called 5 times.

Does samza provide a common data storage for all the tasks? If I use the
KV store which Samza provides , will all the 5 tasks use the same instance
of it?


Regards,
Tushar Mhaskar




On 8/19/15, 12:04 PM, "Yi Pan" <nickpan47@gmail.com> wrote:

>Hi, Mhaskar,
>
>I assume that you were referring to WindowableTask interface, which
>enables
>a basic periodic timer function. As you noticed, it is only a very limited
>version of windowing. For more advanced windowing work, please refer to
>SAMZA-552, which is a WIP now. It will contain the sliding window concept
>with aggregate and join use cases.
>
>For your second question, I can image multiple ways of implementing it:
>
>1) If you have one aggregator job consuming all counting events from all
>upstream counter containers, you can aggregate the counters from all
>containers from the upstream counter job
>2) If you output your single counter result to a distributed KV-store that
>support "compare-and-set", you can always read the value from the store,
>increment the counter, and "compare-and-set" to make sure that you only
>update based on the version that your result is calculated.
>
>There are probably more ways of doing it. I am just putting the above two
>out here as a thought to start with.
>
>Cheers!
>
>-Yi
>
>On Wed, Aug 19, 2015 at 10:07 AM, Mhaskar, Tushar <
>tmhaskar@paypal.com.invalid> wrote:
>
>> Hi,
>>
>>   1.  I came across Windowing functionality in Samza. It looks like it
>> implements a static window.  Is there a sliding window functionality
>> available in Samza?
>>
>>        2.   How to do aggregation across multi-node yarn node, any
>> pointers to it?
>>    E.g :  Lets say I have 2 slave machines where my StreamTask
>> implementation counts all the incoming messages. How and Where can I
>> aggregate the data from the multiple nodes and produce one single count?
>>
>> Regards,
>> Tushar Mhaskar
>>
>>


Mime
View raw message