apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Priyanka Gugale <priya...@datatorrent.com>
Subject Re: Time controlled ingestion operator
Date Tue, 24 May 2016 00:18:33 GMT
Update: This Bandwidth Limiting can be used with any input operator. So you
can apply similar changes to AbstractJDBCPollInputOperator shared by
Devendra earlier.

-Priyanka

On Mon, May 23, 2016 at 4:54 PM, Priyanka Gugale <priyanka@datatorrent.com>
wrote:

> Hi Sunil,
>
> This
> <https://github.com/apache/incubator-apex-malhar/compare/master...DT-Priyanka:jdbc-bandwidth-example?expand=1>
> is the updated JDBCPojoInputOperator to use Bandwidth limiting feature
>  (Just look at JDBCPojoInputOperator changes, other classes are from
> bandwidth PR)
>
> Once you update your code to use extend BandwidthLimitingOperator input
> operator and use BandwidthManager (as done in example code), all you need
> to do is set bandwidth value. In your properties file by adding,
>
>   <property>
>     <name>dt.operator.<OperatorName>.prop.bandwidthManager.bandwidth</name>
>     <value>1073741824</value>
>                                                             // 1 GB (change
> it to desired value)
>   </property>
>
> During hours when you want application to stop reading you can modify
> value of bandwidth to say -1 using dtcli command:
> *set-operator-property operator-name property-name property-value*
>
> You can write cron jobs to change bandwidth property values.
>
> *The right solution would be to update BandwidthManager to accept the
> bandwidth schedule and update consumption accordingly.*
>
> Also please note in this solution the application will still be running
> and operators will be in deployed state, which means using some of your
> cluster resources.
>
> -Priyanka
>
>
> On Mon, May 23, 2016 at 4:27 PM, Sunil Parmar <sparmar@threatmetrix.com>
> wrote:
>
>> Thanks Ram ! That’s the precise reason.
>> Priyanka,
>> Can you elaborate or give more little more guidance to update the
>> application property run time ? How to do that ?
>>
>> -Sunil
>> From: Munagala Ramanath <ram@datatorrent.com>
>> Reply-To: "users@apex.apache.org" <users@apex.apache.org>
>> Date: Monday, May 23, 2016 at 3:42 PM
>> To: "users@apex.apache.org" <users@apex.apache.org>
>> Subject: Re: Time controlled ingestion operator
>>
>> From Sunil's note it sounds like the DB is in production use by other
>> systems during certain hours and wants the Apex based migration to happen
>> at other (non-overlapping) times.
>>
>> Ram
>> On May 23, 2016 2:32 PM, "Priyanka Gugale" <priyanka@datatorrent.com>
>> wrote:
>>
>>> Hi Sunil,
>>>
>>> I don't think we have exactly pause and resume feature in Malhar yet.
>>> But I can think of some other way to achieve this.
>>> We are adding bandwidth control feature to Malhar. The pull request
>>> <https://github.com/apache/incubator-apex-malhar/pull/279> is already
>>> opened for the same. This doesn't have option to configure timed bandwidth
>>> consumption (i.e. use x bandwidth during given time hours), but I think
>>> it's easy to do that in BandwidthManager. Or quick thing is you can try to
>>> update the property "bandwidth" during the time when u don't want operator
>>> to work using some external script (this is short term solution).
>>>
>>> Can you tell the reason why you want to pause the read during certain
>>> time hours?
>>>
>>> -Priyanka
>>>
>>>
>>> On Mon, May 23, 2016 at 2:21 PM, Sunil Parmar <sparmar@threatmetrix.com>
>>> wrote:
>>>
>>>> All,
>>>> We’re writing an app for migration data from legacy JDBC system to
>>>> Kafka. Legacy system as it is still being used in production, we would want
>>>> to pause the JDBC poll input operator for configured hours.
>>>>
>>>> Does malhar ibrary has any operator that we can use for time controlled
>>>> operator ?
>>>> Or is there a way to configure input operators to pause and resume for
>>>> given time ?
>>>>
>>>> -Sunil
>>>>
>>>>
>>>>
>>>
>

Mime
View raw message