apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Wait until setup completes prior to dag execution
Date Thu, 20 Apr 2017 23:27:22 GMT
Additionally that particular operator will not process data till setup is
done. As Pramod pointed out, there is no reason to wait. At worse the
buffer-servers will fill up. Are you worried about the operator code need
to ramp up in processing data and not get a deluge of it?


E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*


On Thu, Apr 20, 2017 at 4:19 PM, Pramod Immaneni <pramod@datatorrent.com>

> Hi,
> I am assuming it is an operator that is not the input operator that has
> the lengthy setup function? Any specific reason you don't want to let data
> in because things don't get committed (like kafka offsets) till all
> operators process the data. If you are worried about input operators are
> going to run-away there is back pressure which will put breaks on when the
> buffers are full.
> Thanks
> On Thu, Apr 20, 2017 at 3:36 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com> wrote:
>> Is there any way to wait for setup to complete prior to allowing data to
>> flow through the DAG? I have an operator with a lengthy setup function so
>> I’m just wondering what the best way to handle this is. My best solution at
>> the moment is to toggle start with an emission rate of zero and toggle it
>> on once the DAG is launched.
>> - Ilya Ganelin
>> [image: id:image001.png@01D1F7A4.F3D42980]
>> ------------------------------
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.

View raw message