apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Parmar <spar...@threatmetrix.com>
Subject Re: Time controlled ingestion operator
Date Tue, 24 May 2016 00:20:31 GMT
Thanks Priyanka !

From: Priyanka Gugale <priyanka@datatorrent.com<mailto:priyanka@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, May 23, 2016 at 5:18 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: Time controlled ingestion operator

Update: This Bandwidth Limiting can be used with any input operator. So you can apply similar
changes to AbstractJDBCPollInputOperator shared by Devendra earlier.


On Mon, May 23, 2016 at 4:54 PM, Priyanka Gugale <priyanka@datatorrent.com<mailto:priyanka@datatorrent.com>>
Hi Sunil,

is the updated JDBCPojoInputOperator to use Bandwidth limiting feature  (Just look at JDBCPojoInputOperator
changes, other classes are from bandwidth PR)

Once you update your code to use extend BandwidthLimitingOperator input operator and use BandwidthManager
(as done in example code), all you need to do is set bandwidth value. In your properties file
by adding,

                                                        // 1 GB (change it to desired value)

During hours when you want application to stop reading you can modify value of bandwidth to
say -1 using dtcli command:
set-operator-property operator-name property-name property-value

You can write cron jobs to change bandwidth property values.

The right solution would be to update BandwidthManager to accept the bandwidth schedule and
update consumption accordingly.

Also please note in this solution the application will still be running and operators will
be in deployed state, which means using some of your cluster resources.


On Mon, May 23, 2016 at 4:27 PM, Sunil Parmar <sparmar@threatmetrix.com<mailto:sparmar@threatmetrix.com>>
Thanks Ram ! That's the precise reason.
Can you elaborate or give more little more guidance to update the application property run
time ? How to do that ?

From: Munagala Ramanath <ram@datatorrent.com<mailto:ram@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, May 23, 2016 at 3:42 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: Time controlled ingestion operator

>From Sunil's note it sounds like the DB is in production use by other systems during certain
hours and wants the Apex based migration to happen at other (non-overlapping) times.


On May 23, 2016 2:32 PM, "Priyanka Gugale" <priyanka@datatorrent.com<mailto:priyanka@datatorrent.com>>
Hi Sunil,

I don't think we have exactly pause and resume feature in Malhar yet. But I can think of some
other way to achieve this.
We are adding bandwidth control feature to Malhar. The pull request<https://github.com/apache/incubator-apex-malhar/pull/279>
is already opened for the same. This doesn't have option to configure timed bandwidth consumption
(i.e. use x bandwidth during given time hours), but I think it's easy to do that in BandwidthManager.
Or quick thing is you can try to update the property "bandwidth" during the time when u don't
want operator to work using some external script (this is short term solution).

Can you tell the reason why you want to pause the read during certain time hours?


On Mon, May 23, 2016 at 2:21 PM, Sunil Parmar <sparmar@threatmetrix.com<mailto:sparmar@threatmetrix.com>>
We're writing an app for migration data from legacy JDBC system to Kafka. Legacy system as
it is still being used in production, we would want to pause the JDBC poll input operator
for configured hours.

Does malhar ibrary has any operator that we can use for time controlled operator ?
Or is there a way to configure input operators to pause and resume for given time ?


View raw message