apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: [malhar-users] Loading the properties file at run time in DT Application.
Date Wed, 30 Dec 2015 16:03:16 GMT
The main feature ask is to change properties at regular interval. That is
achieved via rest api. The dataflow system as a whole will need to keep
track of uploaded properties. The following issues will need to be taken
care of

1. Validation : Correct operator names, logical as well as physical;
correct properties names, run properties annotations, ...
2. Ensure that this upload is part of a re-launch :  Next launch should
remember applied properties
3. Archival : Properties file backed up in a storage (HDFS default) with
some basic versioning (?)

#1 is a one time setup. #2 and #3 are needed to make the application
operable. We will take a look at this ask.

Another big and most likely next ask is to make this transaction ACID. For
a distributed application it gets tricky. There is thread on
dev@apex.incubator.apache.org discussing this in terms of applying operator
instructions in a future streaming window.

Thks,
Amol



Amol


On Wed, Dec 30, 2015 at 6:01 AM, Sandeep Deshmukh <sandeep@datatorrent.com>
wrote:

> For 3, handleIdleTime() is not the right thing for this requirement. This
> function call is not guaranteed. You can use beginWindow or endWindow for
> such purpose.
>
> Regards,
> Sandeep
>
> On Wed, Dec 30, 2015 at 6:02 PM, Yogi Devendra <devendra@datatorrent.com>
> wrote:
>
>> 1. Web UI has facility to manually change the operator properties.
>>
>> 2. If you need to change it programmatically you can have a look at REST
>> API documentation available here:
>> http://docs.datatorrent.com/dtgateway_api/
>>
>> This is like 'push'  the properties whenever there is some change.
>>
>> 3. If you are looking for continuous polling of some properties file
>> where some external program is writing property values:
>> You can make use of handleIdleTime() from in your operator code and
>> achieve this.
>>
>> But, I would recommend option 2 over option 3.
>>
>> ~ Yogi
>>
>> On Wed, Dec 30, 2015 at 5:20 PM, PULLARAO KOTA <10fe1a0571@gmail.com>
>> wrote:
>>
>>> Hi I am not using Dtcli,Using the DataTorrent Console Web UI To Upload
>>> Jar and Run the Application i want to load the properties at the time of
>>> Launching the application instead of giving each property separately.
>>>
>>> The Properties file will Be As Key=Value pairs.Please Suggest a method
>>> to solve the problem.
>>>
>>>
>>>
>>> On Wednesday, December 30, 2015 at 4:32:35 PM UTC+5:30, Yogi Devendra
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I assume you are referring to operator properties loaded from xml or
>>>> json configuration.
>>>>
>>>> 1. dtcli provides a mechanism to connect to the running application
>>>> using following command:
>>>>
>>>> connect app-id
>>>> Connect to an app
>>>>
>>>> 2. Following command is available after you connect the application to
>>>> set the operator properties.
>>>>
>>>> set-operator-property operator-name property-name property-value
>>>> Set a property of an operator
>>>>
>>>> ~ Yogi
>>>>
>>>> P. S: malhar...@googlegroups.com is now deprecated.
>>>>  Kindly use us...@apex.incubator.apache.org mailing list for any
>>>> further  questions.
>>>>
>>>> On Wed, Dec 30, 2015 at 3:04 PM, PULLARAO KOTA <10fe1...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I have a DataTorrent application which loads the properties from file
>>>>> and use them in code, and some times we have to change the properties
at
>>>>> regular intervals, in this case every time we have to stop and start
the
>>>>> Application. This will be a hectic job restart the Application each time.
>>>>> Is there any method how to read properties at run time even after we
change
>>>>> the file?
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Malhar" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to malhar-users...@googlegroups.com.
>>>>> To post to this group, send email to malhar...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/malhar-users.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Malhar" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to malhar-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to malhar-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/malhar-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Malhar" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to malhar-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to malhar-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/malhar-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Malhar" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to malhar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to malhar-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/malhar-users.
> For more options, visit https://groups.google.com/d/optout.
>

Mime
View raw message