apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "York, Brennon" <Brennon.Y...@capitalone.com>
Subject Re: Throttling of `emitTuples`?
Date Tue, 01 Sep 2015 00:27:24 GMT
Definitely the former for the use case. Not that the downstream operators
can¹t (eventually) keep up, but we¹re building up a demo application where
(in prod) we¹ll only receive a set number of events per second. Right now,
since we aren¹t in prod, we¹re hooking up an operator to read a file from
HDFS, but it goes *far* beyond what is necessary as far as tuples per
second is concerned. We figured (since we¹re using the
AbstractInputOperator) we could throttle the batch size to the amount we
want per second and then set the `emitTuples` call to only be called once
per second thus giving us a down-throttled (and guaranteed) events per
second number to continue testing with. Does that make sense?

@Chetan thanks for the info! It¹s sounding like the reality is that its up
to us as operator writers to check when `emitTuples` is called and, for
instance, if its been more than a second, call it again. Is that correct?

On 8/31/15, 5:19 PM, "Vlad Rozov" <v.rozov@datatorrent.com> wrote:

>Do you want to slow down an input operator so downstream operators can
>keep up with it or in your case input operator puts too much pressure on
>external data source, so is the need for throttling? I may be wrong, but
>I think there is no such property and I am curious to see what will be
>the use case for the property.
>
>Thank you,
>
>Vlad
>
>On 8/31/15 16:25, York, Brennon wrote:
>> Hey all, is there a property out there that throttles the `emitTuples`
>>call for input operators? I¹ve been hunting down various properties and
>>can¹t seem to find it for the life of me. I¹m sure I¹m missing something
>>simpleŠ
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>>proprietary to Capital One and/or its affiliates and may only be used
>>solely in performance of work or services for Capital One. The
>>information transmitted herewith is intended only for use by the
>>individual or entity to which it is addressed. If the reader of this
>>message is not the intended recipient, you are hereby notified that any
>>review, retransmission, dissemination, distribution, copying or other
>>use of, or taking of any action in reliance upon this information is
>>strictly prohibited. If you have received this communication in error,
>>please contact the sender and delete the material from your computer.
>>
>

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.


Mime
View raw message