falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satish Mittal <satish.mit...@inmobi.com>
Subject Re: On configuring two source clusters due to colo requirement
Date Tue, 15 Jul 2014 04:34:16 GMT

Given that both ETL clusters are producing the same data-set independent of
each other and the aim is to replicate the data-set within colo (to avoid
any cross-colo data movement), you could simply have 2 instances of the
same feed, one per colo:

<cluster name=“colo1ETL type="source">
<cluster name=“colo1A” type="target">

<cluster name=“colo2ETL type="source">
<cluster name=“colo2A” type="target">

The 1st error was coming since multiple source replication was configured
(which needs partition expressions to be specified). Also that
configuration would have ended up moving data across colos, which is
against your desired goal.


On Mon, Jul 14, 2014 at 11:52 PM, John Yu <johnyu0520@gmail.com> wrote:

> Hey all,
> We currently have the following use case:
> Colo1 has 1 ETL cluster (Colo1-ETL) and 1 adhoc cluster (Colo1-A)
> Colo2 has 1 ETL cluster (Colo2-ETL) and 1 adhoc cluster (Colo2-A)
> Due to the bandwidth constraint between the two colo's, we are thinking of
> having the 2 ETL clusters perform the same computation to generate the same
> dataset, and have the 2 adhoc clusters pull from their respective
> colo-local ETL cluster.
> What would be a good way to configure this feed?
> I've tried the following:
> <cluster name=“colo1ETL type="source">
> <cluster name="colo2ETL" type="source">
> <cluster name=“colo1A” type="target">
> <cluster name="colo2A” type="target">
> Error: Partition expression has to be specified for cluster colo1ETL as
> there are more than one source clusters
> <cluster name=“colo1ETL”>
> <cluster name="colo2ETL”>
> <cluster name=“colo1A” type="target">
> <cluster name="colo2A” type="target">
> Error: Feed: pve-intermediate should have atleast one source cluster
> defined
> Thanks!
> John

The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message