Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of johnyu0520@gmail.com designates
 209.85.160.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPFf3CosFYWXDkExa=uDd9O47kBGZ0UGFJEOrAh3N5PkxXnjyw@mail.gmail.com>
References: 
 <CAJQpoBSnEyhXYRxGZd_76qNf09DTr1v-hRCpzJzap12uEMC_Bw@mail.gmail.com>
 <CAPFf3CosFYWXDkExa=uDd9O47kBGZ0UGFJEOrAh3N5PkxXnjyw@mail.gmail.com>
From: John Yu <johnyu0520@gmail.com>
Date: Tue, 15 Jul 2014 12:30:51 -0700
Message-ID: 
 <CAJQpoBT=J4QD5B2fzf7PTT2tMP93KH8gh5GFSyaJ67KHrY5Yaw@mail.gmail.com>
Subject: Re: On configuring two source clusters due to colo requirement
To: dev@falcon.incubator.apache.org
Cc: Venkat R <veramacha@yahoo.com>,
 Seetharam Venkatesh <venkatesh@innerzeal.com>
Content-Type: multipart/alternative; boundary=001a11332a2ed77e3a04fe407337

--001a11332a2ed77e3a04fe407337
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hey Satish,

Thanks for your reply!

I can see how setting up that way would definitely work.
Also, it is probably technically more correct as well, as data generated by
different processes should be considered different.

However, we are thinking along the lines of data discovery, in which a
critical dataset might be computed on different colos simultaneously for
both DR and load balancing purposes.  In this scenario, we would somehow
like the end users to know that feed1 and feed2 are logically the same
data, and they are free to pick one to use.

Just wondering whether it make sense to support multiple sources and
multiple targets without specifying partition (and maybe the target cluster
have to specify the order of sources from which to copy).  Also I am
guessing that this "multiple sources and multiple targets without
specifying partition" requirement must have came up before, and what was
the thought process that went behind not supporting it in the end.

Thanks a lot!
John


2014-07-14 21:34 GMT-07:00 Satish Mittal <satish.mittal@inmobi.com>:

> Hi,
>
> Given that both ETL clusters are producing the same data-set independent =
of
> each other and the aim is to replicate the data-set within colo (to avoid
> any cross-colo data movement), you could simply have 2 instances of the
> same feed, one per colo:
>
> feed1:
> <cluster name=3D=E2=80=9Ccolo1ETL type=3D"source">
> <cluster name=3D=E2=80=9Ccolo1A=E2=80=9D type=3D"target">
>
> feed2:
> <cluster name=3D=E2=80=9Ccolo2ETL type=3D"source">
> <cluster name=3D=E2=80=9Ccolo2A=E2=80=9D type=3D"target">
>
> The 1st error was coming since multiple source replication was configured
> (which needs partition expressions to be specified). Also that
> configuration would have ended up moving data across colos, which is
> against your desired goal.
>
> Thanks,
> Satish
>
>
> On Mon, Jul 14, 2014 at 11:52 PM, John Yu <johnyu0520@gmail.com> wrote:
>
> > Hey all,
> >
> > We currently have the following use case:
> > Colo1 has 1 ETL cluster (Colo1-ETL) and 1 adhoc cluster (Colo1-A)
> > Colo2 has 1 ETL cluster (Colo2-ETL) and 1 adhoc cluster (Colo2-A)
> >
> > Due to the bandwidth constraint between the two colo's, we are thinking
> of
> > having the 2 ETL clusters perform the same computation to generate the
> same
> > dataset, and have the 2 adhoc clusters pull from their respective
> > colo-local ETL cluster.
> >
> > What would be a good way to configure this feed?
> >
> > I've tried the following:
> > <cluster name=3D=E2=80=9Ccolo1ETL type=3D"source">
> > <cluster name=3D"colo2ETL" type=3D"source">
> > <cluster name=3D=E2=80=9Ccolo1A=E2=80=9D type=3D"target">
> > <cluster name=3D"colo2A=E2=80=9D type=3D"target">
> > Error: Partition expression has to be specified for cluster colo1ETL as
> > there are more than one source clusters
> >
> > <cluster name=3D=E2=80=9Ccolo1ETL=E2=80=9D>
> > <cluster name=3D"colo2ETL=E2=80=9D>
> > <cluster name=3D=E2=80=9Ccolo1A=E2=80=9D type=3D"target">
> > <cluster name=3D"colo2A=E2=80=9D type=3D"target">
> > Error: Feed: pve-intermediate should have atleast one source cluster
> > defined
> >
> >
> > Thanks!
> >
> > John
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for th=
e
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileg=
ed
> information. If you are not the intended recipient you are hereby notifie=
d
> that any disclosure, copying, distribution or taking any action in relian=
ce
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmissi=
on
> of the information contained in this communication nor for any delay in i=
ts
> receipt.
>


--=20
=E4=BD=99=E5=AE=88=E4=B8=AD  John Yu (Yu, Shoou-Jong)
Mobile: 650-691-3314

--001a11332a2ed77e3a04fe407337--