heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanjee...@gmail.com>
Subject Re: Hello heron community
Date Mon, 30 Oct 2017 22:07:10 GMT
My initial hunch is that these are different goals and therefore concur
with your thought that you could do this in phase 2.

On Mon, Oct 30, 2017 at 3:03 PM, Saikat Kanjilal <sxk1969@hotmail.com>
wrote:

> The goal is to tie together spark streaming and heron, I could very easily
> see an architecture where we build topologies in heron to apply business
> logic and pump them into time windows with spark streaming where can apply
> machine learning algorithms inside these windows.
>
>
> Detailed Example:
>
>
> Stock Trading or gamin use case:
>
>
> Heron is used to extract real time data from a gaming or trading client,
>
> Heron will apply a set of business rules to cleanup this data to fit into
> a set of machine learning models
>
> We need to then figure out how to connect spark streaming to heron, in the
> case of spark streaming it listens to some tcp or http endpoint so the
> objective would be to figure out the bridge between the two worlds.
>
>
> My point here is that if I am going through and building out a scala API
> it might make sense to also build a connector to a tech stack entirely
> written using scala as a first class citizen.
>
>
>
> Let me know your thoughts, we can also push this to phase 2 once the heron
> scala api is readily available.
>
>
> ________________________________
> From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> Sent: Monday, October 30, 2017 2:54 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I didn't really get what you meant by 'stream endpoint would be coming from
> heron'. Could you please elaborate? Preferably with an example?
>
> On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> wrote:
>
> > I'm conflicted a bit on 1, here's why I feel like most of our users would
> > want to use the spouts/bolts API who are coming over from Storm or using
> > heron for the first time, however the Streamlets interface seems like it
> > has the right level of abstraction, I think I will start with the
> Streamlet
> > interface and add in the spout/bolt API as needed.   One other thought I
> > had was to figure out an integration plan with spark, to this end the fit
> > between heron and spark would be in using spark streaming where the
> stream
> > endpoint would be coming from heron, not sure if that should be part of
> > this effort or not, what do you guys think?
> >
> >
> > ________________________________
> > From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> > Sent: Monday, October 30, 2017 2:17 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > I have a couple of questions
> > 1. Do you plan on exposing both the low level(spouts/bolts) and the
> > streamlet api? Or are you preferring one over another?
> > 2. My suggestion would be to start with the Streamlet interface.
> Primarily
> > because a) Most of scala ml libraries that I've seen tend to operate on
> the
> > streamlet kind of interface and b) It has a far smaller surface area(i.e.
> > number of interfaces) so might be easy to get something up quickly for
> > testing.
> > Thanks!
> >
> > On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> > wrote:
> >
> > > Thanks Sanjeev,  I had an initial idea that I wanted to float on the
> > list,
> > > I was thinking that as part of the initial scala port of the API I'd
> like
> > > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > >
> > > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > > javatoscala.com
> > > How does it work? Java to Scala converter is created with Play
> framework
> > > and Scalagen library. I don't want you to see my code. No problem. Java
> > to
> > > Scala converter ...
> > >
> > >
> > > I propose that we use the above tool to do an initial conversion of all
> > > the Java functional and oo interfaces and then plug that into the
> initial
> > > design doc as well as the codebase to get things off the ground.   I
> have
> > > already forked the heron repo and will be doing this off of a child
> > branch
> > > based on my fork.
> > >
> > >
> > > How does that sound to folks, anyone has strong objections, of course I
> > > fully realize that I will still need to do a lot of work around
> > refactoring
> > > the code even when using the converter so I'm prepared for that, the
> goal
> > > in using the converters is to save a bit of time so that manual
> > > intervention isnt necessarily needed in designing every interface.
> > >
> > > Thoughts.
> > >
> > > ________________________________
> > > From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> > > Sent: Sunday, October 29, 2017 10:18 PM
> > > To: dev@heron.incubator.apache.org
> > > Subject: Re: Hello heron community
> > >
> > > Hi Saikat,
> > > Welcome to the Heron users group. Your plan of action sounds good to
> me.
> > A
> > > native scala interface would be great for all those scala fans. Eagerly
> > > awaiting design doc.
> > > Thanks!
> > >
> > > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> > > wrote:
> > >
> > > > Hello Folks,
> > > >
> > > > I'm interested in taking and driving the following github component
> > (this
> > > > means design/development/architecture and more).
> > > >
> > > >
> > > > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > Support for Scala Instance and APIs.
> > >
> > >
> > >
> > > >
> > > >
> > > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > > > github.com/twitter/heron/issues/668>
> > > >
> > > > Ability to write native scala topologies · Issue #668 ...<
> > > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > Support for Scala Instance and APIs.
> > >
> > >
> > >
> > > > github.com
> > > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > > processing
> > > > engine from Twitter
> > > >
> > > >
> > > >
> > > >
> > > > I've joined the heron slack channel as recommended and am ready to
> > start
> > > > by creating a master JIRA issue to track this if not already done, I
> > > wanted
> > > > some guidance from a committer or two to help guide me a bit take
> care
> > of
> > > > at he administrative details and how to get this kicked off the
> ground.
> > > My
> > > > background includes working on apache reef codebase as well as
> dabbling
> > > in
> > > > apache mahout and apache flume as well as storm and various
> nosql/graph
> > > > databases.  Looking forward to being part of the community.
> > > >
> > > >
> > > >
> > > > The rough plan of action in my mind so you know:
> > > >
> > > > 1) Review heron architecture and codebase in detail
> > > >
> > > > 2) Create an umbrella JIRA with the first series of tasks
> > > >
> > > > 3) Come up with a design doc
> > > >
> > > > 4) Build the interfaces
> > > >
> > > > 5) Add more dev tasks to JIRA
> > > >
> > > > 6) Drive the implementation
> > > >
> > > >
> > > > Thanks in advance for your help.
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message