Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Received-SPF: pass (nike.apache.org: domain of sriksun@hotmail.com designates
 65.55.111.173 as permitted sender)
Message-ID: <BLU179-W84C82B33B22EAD229D8872A4220@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_5cb3a6ab-7950-4054-b7c8-850d4b6c2319_"
From: Srikanth Sundarrajan <sriksun@hotmail.com>
To: "dev@falcon.apache.org" <dev@falcon.apache.org>
Subject: RE: Streaming Feed
Date: Thu, 12 Feb 2015 21:20:57 +0530
Importance: Normal
In-Reply-To: 
 <CAPqfsdztNbCJMkNmVHFt4FNag25W0+cTdgWDtb+uFV27Np4aOQ@mail.gmail.com>
References: 
 <CAPqfsdwjvqYpUs88ooa_80WswjfRnmzaMJH=NqGxzxuEohKpXQ@mail.gmail.com>,<54DB1B33.5010500@nanthrax.net>,<CAPqfsdztNbCJMkNmVHFt4FNag25W0+cTdgWDtb+uFV27Np4aOQ@mail.gmail.com>
MIME-Version: 1.0

--_5cb3a6ab-7950-4054-b7c8-850d4b6c2319_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

This is been an idea lingering in my mind for a while. I will be very suppo=
rtive of any effort to create a stream abstract similar in lines with feed =
(or this may not be required=2C if we do a major overhaul with respect to o=
rchestration in falcon=2C where tight requirement of feed having a frequenc=
y is done away with) and have process work with these streams. In which cas=
e the orchestration should happen through Nimbus or Spark Master instead of=
 Oozie.

In other words:
* Feed/Stream to be a primitive entity in falcon which declares that there =
is a continuous flow of data as per schema and is not bound to any arrival =
periodicity
* Replication/Mirroring on this would essentially use standard data transpo=
rt mechanisms to ship data also on a streaming fashion
* Processes that are defined over these continuous streams are to be orches=
trated over an appropriate engine such as Nimbus (in case of Storm) or simi=
lar system. Processes that are defined in this way also doesn't have period=
icity and are continuous.=20

This topic requires more conversation before we figure the way forward. Am =
assuming=2C more than one of us are thinking about this.

Regards
Srikanth Sundarrajan

> Date: Wed=2C 11 Feb 2015 15:30:47 +0530
> Subject: Re: Streaming Feed
> From: sharad@apache.org
> To: dev@falcon.apache.org
>=20
> Thanks Jean=2C this will be quite useful. I am wondering if this will req=
uire
> a new partitioning construct in the feed as well like micro-batches=2C et=
c.
>=20
> Sharad
>=20
> On Wed=2C Feb 11=2C 2015 at 2:34 PM=2C Jean-Baptiste Onofr=E9 <jb@nanthra=
x.net>
> wrote:
>=20
> > Hi Sharad=2C
> >
> > I sent an e-mail last week about support of Spark (SparkStreaming) in
> > workflow/process. It's basically very close to what you propose.
> >
> > IMHO=2C it should be a new impl of workflow or at least the support of =
a new
> > kind of processes (it's what I have in mind).
> >
> > Regards
> > JB
> >
> >
> > On 02/11/2015 09:38 AM=2C Sharad Agarwal wrote:
> >
> >> I am looking for a generic schema aware feed construct for streaming
> >> workflow. The schema can be managed by a catalog service like HCatalog=
.
> >> The
> >> streaming workflow executor would be a system like
> >> Storm/SparkStreaming/Samza.
> >>
> >> I want to know if this is the right thing to be supported in Falcon an=
d if
> >> yes what is the plugging interface for that. Would this be a new
> >> implementation of workflow engine ?
> >>
> >> Thanks
> >> Sharad
> >>
> >>
> > --
> > Jean-Baptiste Onofr=E9
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
 		 	   		  =

--_5cb3a6ab-7950-4054-b7c8-850d4b6c2319_--