Mailing-List: contact dev-help@heron.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@heron.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CY4PR1401MB202175C774DC63D6DBE74F33A8590@CY4PR1401MB2021.namprd14.prod.outlook.com>
References: <CY4PR1401MB202185FCBE1248FAC0CCB608A8590@CY4PR1401MB2021.namprd14.prod.outlook.com>
 <CANx6OK908ucPihwfy6RrJLfNCYsVmzgg6VxvJ4oagHSzX4rTMA@mail.gmail.com>
 <CY4PR1401MB202137977CD0FDFB98FD4D50A8590@CY4PR1401MB2021.namprd14.prod.outlook.com>
 <CANx6OK_xNNTz+jThmnujNLuBPnvuOwGvQq=FHeqS=EOhXDzUyA@mail.gmail.com>
 <CY4PR1401MB2021CE279E3D6D8C24FAFCB3A8590@CY4PR1401MB2021.namprd14.prod.outlook.com>
 <CANx6OK-3dsOZsQtudvveiHSxbFVSrFdAiSn4HCq-nCB4dsaEaQ@mail.gmail.com> <CY4PR1401MB202175C774DC63D6DBE74F33A8590@CY4PR1401MB2021.namprd14.prod.outlook.com>
From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
Date: Mon, 30 Oct 2017 15:07:10 -0700
Message-ID: <CANx6OK97fy=sPVgATAre9VU9vPP4j_UAZPeuhM1WVNNRrh5yTA@mail.gmail.com>
Subject: Re: Hello heron community
To: dev@heron.incubator.apache.org
Content-Type: multipart/alternative; boundary="94eb2c184a663a077e055ccadc0f"
archived-at: Mon, 30 Oct 2017 22:07:22 -0000

--94eb2c184a663a077e055ccadc0f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

My initial hunch is that these are different goals and therefore concur
with your thought that you could do this in phase 2.

On Mon, Oct 30, 2017 at 3:03 PM, Saikat Kanjilal <sxk1969@hotmail.com>
wrote:

> The goal is to tie together spark streaming and heron, I could very easil=
y
> see an architecture where we build topologies in heron to apply business
> logic and pump them into time windows with spark streaming where can appl=
y
> machine learning algorithms inside these windows.
>
>
> Detailed Example:
>
>
> Stock Trading or gamin use case:
>
>
> Heron is used to extract real time data from a gaming or trading client,
>
> Heron will apply a set of business rules to cleanup this data to fit into
> a set of machine learning models
>
> We need to then figure out how to connect spark streaming to heron, in th=
e
> case of spark streaming it listens to some tcp or http endpoint so the
> objective would be to figure out the bridge between the two worlds.
>
>
> My point here is that if I am going through and building out a scala API
> it might make sense to also build a connector to a tech stack entirely
> written using scala as a first class citizen.
>
>
>
> Let me know your thoughts, we can also push this to phase 2 once the hero=
n
> scala api is readily available.
>
>
> ________________________________
> From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> Sent: Monday, October 30, 2017 2:54 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I didn't really get what you meant by 'stream endpoint would be coming fr=
om
> heron'. Could you please elaborate? Preferably with an example?
>
> On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> wrote:
>
> > I'm conflicted a bit on 1, here's why I feel like most of our users wou=
ld
> > want to use the spouts/bolts API who are coming over from Storm or usin=
g
> > heron for the first time, however the Streamlets interface seems like i=
t
> > has the right level of abstraction, I think I will start with the
> Streamlet
> > interface and add in the spout/bolt API as needed.   One other thought =
I
> > had was to figure out an integration plan with spark, to this end the f=
it
> > between heron and spark would be in using spark streaming where the
> stream
> > endpoint would be coming from heron, not sure if that should be part of
> > this effort or not, what do you guys think?
> >
> >
> > ________________________________
> > From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> > Sent: Monday, October 30, 2017 2:17 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > I have a couple of questions
> > 1. Do you plan on exposing both the low level(spouts/bolts) and the
> > streamlet api? Or are you preferring one over another?
> > 2. My suggestion would be to start with the Streamlet interface.
> Primarily
> > because a) Most of scala ml libraries that I've seen tend to operate on
> the
> > streamlet kind of interface and b) It has a far smaller surface area(i.=
e.
> > number of interfaces) so might be easy to get something up quickly for
> > testing.
> > Thanks!
> >
> > On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> > wrote:
> >
> > > Thanks Sanjeev,  I had an initial idea that I wanted to float on the
> > list,
> > > I was thinking that as part of the initial scala port of the API I'd
> like
> > > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java t=
o
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java t=
o
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framewor=
k
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > >
> > > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java t=
o
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java t=
o
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framewor=
k
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > > javatoscala.com
> > > How does it work? Java to Scala converter is created with Play
> framework
> > > and Scalagen library. I don't want you to see my code. No problem. Ja=
va
> > to
> > > Scala converter ...
> > >
> > >
> > > I propose that we use the above tool to do an initial conversion of a=
ll
> > > the Java functional and oo interfaces and then plug that into the
> initial
> > > design doc as well as the codebase to get things off the ground.   I
> have
> > > already forked the heron repo and will be doing this off of a child
> > branch
> > > based on my fork.
> > >
> > >
> > > How does that sound to folks, anyone has strong objections, of course=
 I
> > > fully realize that I will still need to do a lot of work around
> > refactoring
> > > the code even when using the converter so I'm prepared for that, the
> goal
> > > in using the converters is to save a bit of time so that manual
> > > intervention isnt necessarily needed in designing every interface.
> > >
> > > Thoughts.
> > >
> > > ________________________________
> > > From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> > > Sent: Sunday, October 29, 2017 10:18 PM
> > > To: dev@heron.incubator.apache.org
> > > Subject: Re: Hello heron community
> > >
> > > Hi Saikat,
> > > Welcome to the Heron users group. Your plan of action sounds good to
> me.
> > A
> > > native scala interface would be great for all those scala fans. Eager=
ly
> > > awaiting design doc.
> > > Thanks!
> > >
> > > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1969@hotmail.com=
>
> > > wrote:
> > >
> > > > Hello Folks,
> > > >
> > > > I'm interested in taking and driving the following github component
> > (this
> > > > means design/development/architecture and more).
> > > >
> > > >
> > > > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > > [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https=
://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twi=
tter/heron<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https:/=
/
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitt=
er/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > Support for Scala Instance and APIs.
> > >
> > >
> > >
> > > >
> > > >
> > > > [https://avatars1.githubusercontent.com/u/367684?v=3D4&s=3D400]<htt=
ps://
> > > > github.com/twitter/heron/issues/668>
> > > >
> > > > Ability to write native scala topologies =C2=B7 Issue #668 ...<
> > > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https:/=
/
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitt=
er/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https=
://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twi=
tter/heron<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https:/=
/
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitt=
er/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=3D400&v=3D4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies =C2=B7 Issue #668 =C2=B7 twitter=
/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > Support for Scala Instance and APIs.
> > >
> > >
> > >
> > > > github.com
> > > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > > processing
> > > > engine from Twitter
> > > >
> > > >
> > > >
> > > >
> > > > I've joined the heron slack channel as recommended and am ready to
> > start
> > > > by creating a master JIRA issue to track this if not already done, =
I
> > > wanted
> > > > some guidance from a committer or two to help guide me a bit take
> care
> > of
> > > > at he administrative details and how to get this kicked off the
> ground.
> > > My
> > > > background includes working on apache reef codebase as well as
> dabbling
> > > in
> > > > apache mahout and apache flume as well as storm and various
> nosql/graph
> > > > databases.  Looking forward to being part of the community.
> > > >
> > > >
> > > >
> > > > The rough plan of action in my mind so you know:
> > > >
> > > > 1) Review heron architecture and codebase in detail
> > > >
> > > > 2) Create an umbrella JIRA with the first series of tasks
> > > >
> > > > 3) Come up with a design doc
> > > >
> > > > 4) Build the interfaces
> > > >
> > > > 5) Add more dev tasks to JIRA
> > > >
> > > > 6) Drive the implementation
> > > >
> > > >
> > > > Thanks in advance for your help.
> > > >
> > > >
> > >
> >
>

--94eb2c184a663a077e055ccadc0f--