apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Is there a way to schedule an operator?
Date Tue, 13 Jun 2017 23:01:48 GMT
It is one way to go, to have an operator implement the scheduling. You can
signal the operator using the change of a property. Look at the
set-operator-property command in the apex cli, it can be used to change a
property which will result in the corresponding setter method being called
in the operator and you can use that as a signal to do further operations.

On Tue, Jun 13, 2017 at 3:30 PM, Guilherme Hott <guilhermehott@gmail.com>
wrote:

> I was thinking about to use a cron sending a kafka message and in my DAG
> I'll have a kafka input operator to consume this message and start the
> process. I think this work but I would like to know if have something more
> appropriate.
>
> On Tue, Jun 13, 2017 at 3:25 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
>> There is no built scheduler to schedule the DAGs at a prescribed time,
>> you would need to use some external mechanisms. Because it is a daily
>> one-time activity, would something like cron work for you?
>>
>> On Tue, Jun 13, 2017 at 3:22 PM, Guilherme Hott <guilhermehott@gmail.com>
>> wrote:
>>
>>> Because I am syncing my data from a table in a database to HDFS and I
>>> want to do this just once a day to save processing use.
>>>
>>> On Tue, Jun 13, 2017 at 2:45 PM, Ganelin, Ilya <
>>> Ilya.Ganelin@capitalone.com> wrote:
>>>
>>>> Why don’t you want your dag to continue running? Are there resources
>>>> you wish to release?
>>>>
>>>>
>>>>
>>>> - Ilya Ganelin
>>>>
>>>> [image: id:image001.png@01D1F7A4.F3D42980]
>>>>
>>>>
>>>>
>>>> *From: *Guilherme Hott <guilhermehott@gmail.com>
>>>> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
>>>> *Date: *Tuesday, June 13, 2017 at 2:29 PM
>>>> *To: *"users@apex.apache.org" <users@apex.apache.org>
>>>> *Subject: *Is there a way to schedule an operator?
>>>>
>>>>
>>>>
>>>> Hi guys,
>>>>
>>>>
>>>>
>>>> Is there a way to schedule an operator? I need an operator start the
>>>> DAG once a day at 00am.
>>>>
>>>>
>>>>
>>>> Best
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Guilherme Hott*
>>>>
>>>> *Software Engineer*
>>>>
>>>> Skype: guilhermehott
>>>>
>>>> @guilhermehott
>>>>
>>>> https://www.linkedin.com/in/guilhermehott
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> The information contained in this e-mail is confidential and/or
>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>> solely in performance of work or services for Capital One. The information
>>>> transmitted herewith is intended only for use by the individual or entity
>>>> to which it is addressed. If the reader of this message is not the intended
>>>> recipient, you are hereby notified that any review, retransmission,
>>>> dissemination, distribution, copying or other use of, or taking of any
>>>> action in reliance upon this information is strictly prohibited. If you
>>>> have received this communication in error, please contact the sender and
>>>> delete the material from your computer.
>>>>
>>>
>>>
>>>
>>> --
>>> *Guilherme Hott*
>>> *Software Engineer*
>>> Skype: guilhermehott
>>> @guilhermehott
>>> https://www.linkedin.com/in/guilhermehott
>>>
>>>
>>
>
>
> --
> *Guilherme Hott*
> *Software Engineer*
> Skype: guilhermehott
> @guilhermehott
> https://www.linkedin.com/in/guilhermehott
>
>

Mime
View raw message