heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanjee...@gmail.com>
Subject Re: Hello heron community
Date Mon, 30 Oct 2017 21:54:47 GMT
I didn't really get what you meant by 'stream endpoint would be coming from
heron'. Could you please elaborate? Preferably with an example?

On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1969@hotmail.com>
wrote:

> I'm conflicted a bit on 1, here's why I feel like most of our users would
> want to use the spouts/bolts API who are coming over from Storm or using
> heron for the first time, however the Streamlets interface seems like it
> has the right level of abstraction, I think I will start with the Streamlet
> interface and add in the spout/bolt API as needed.   One other thought I
> had was to figure out an integration plan with spark, to this end the fit
> between heron and spark would be in using spark streaming where the stream
> endpoint would be coming from heron, not sure if that should be part of
> this effort or not, what do you guys think?
>
>
> ________________________________
> From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> Sent: Monday, October 30, 2017 2:17 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I have a couple of questions
> 1. Do you plan on exposing both the low level(spouts/bolts) and the
> streamlet api? Or are you preferring one over another?
> 2. My suggestion would be to start with the Streamlet interface. Primarily
> because a) Most of scala ml libraries that I've seen tend to operate on the
> streamlet kind of interface and b) It has a far smaller surface area(i.e.
> number of interfaces) so might be easy to get something up quickly for
> testing.
> Thanks!
>
> On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> wrote:
>
> > Thanks Sanjeev,  I had an initial idea that I wanted to float on the
> list,
> > I was thinking that as part of the initial scala port of the API I'd like
> > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> >
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> > I propose that we use the above tool to do an initial conversion of all
> > the Java functional and oo interfaces and then plug that into the initial
> > design doc as well as the codebase to get things off the ground.   I have
> > already forked the heron repo and will be doing this off of a child
> branch
> > based on my fork.
> >
> >
> > How does that sound to folks, anyone has strong objections, of course I
> > fully realize that I will still need to do a lot of work around
> refactoring
> > the code even when using the converter so I'm prepared for that, the goal
> > in using the converters is to save a bit of time so that manual
> > intervention isnt necessarily needed in designing every interface.
> >
> > Thoughts.
> >
> > ________________________________
> > From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> > Sent: Sunday, October 29, 2017 10:18 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > Hi Saikat,
> > Welcome to the Heron users group. Your plan of action sounds good to me.
> A
> > native scala interface would be great for all those scala fans. Eagerly
> > awaiting design doc.
> > Thanks!
> >
> > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> > wrote:
> >
> > > Hello Folks,
> > >
> > > I'm interested in taking and driving the following github component
> (this
> > > means design/development/architecture and more).
> > >
> > >
> > > https://github.com/twitter/heron/issues/668
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > >
> > >
> > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 ...<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > processing
> > > engine from Twitter
> > >
> > >
> > >
> > >
> > > I've joined the heron slack channel as recommended and am ready to
> start
> > > by creating a master JIRA issue to track this if not already done, I
> > wanted
> > > some guidance from a committer or two to help guide me a bit take care
> of
> > > at he administrative details and how to get this kicked off the ground.
> > My
> > > background includes working on apache reef codebase as well as dabbling
> > in
> > > apache mahout and apache flume as well as storm and various nosql/graph
> > > databases.  Looking forward to being part of the community.
> > >
> > >
> > >
> > > The rough plan of action in my mind so you know:
> > >
> > > 1) Review heron architecture and codebase in detail
> > >
> > > 2) Create an umbrella JIRA with the first series of tasks
> > >
> > > 3) Come up with a design doc
> > >
> > > 4) Build the interfaces
> > >
> > > 5) Add more dev tasks to JIRA
> > >
> > > 6) Drive the implementation
> > >
> > >
> > > Thanks in advance for your help.
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message